4 CVRANAL - Coverage Analysis

4.1 Introduction

CVRANAL reads the coverage (.cvr) files written by the test version of a program and writes a report showing which code blocks have never been executed in the current series of tests, and which have been executed most frequently (execution hot-spots). Based on this report, it should be possible to devise additional tests to reduce the amount of untested code. The analysis of execution hot-spots may also suggest ways to make the program run faster.

CVRANAL is also able to merge the execution frequency data with the original source code (or a copy thereof). The number of times each block has been executed is inserted as an end-of-line ('!') comment, starting at a specified column (e.g. 133). The comment is suppressed on lines which extend beyond the specified column.

Coverage Analysis is an important tool for anyone involved in software quality assurance. Clearly a program cannot be said to have been properly tested if there are parts of the source code which have never been executed, and programmers are often surprised to find that even quite extensive testing can leave large amounts of untested code. Any thorough QA procedure should use coverage analysis in conjunction with both static and dynamic analysis, since each approach will find problems that the others cannot detect.

4.2 The CVRANAL Command Line

To start a run, simply type cvranal followed by the name of the coverage file(s) you wish to process. You can use wild-cards to specify more than one file. For example:

      cvranal *.cvr

processes the contents of all .cvr files in the current directory.

Under Windows, .cvr is assumed if no extension is specified (b* is treated as b*.cvr). The default file name is *.cvr. Thus, if you just type cvranal, all .cvr files in the current directory are processed.

On Linux and Mac, the extension must be specified, and there is no default file name.

The operation of CVRANAL may be modified by command line switches. Currently the following switches are available:

For example

      cvranal a*.cvr to=cvrfred.out fig=cvrfred.fig  513=0

This command causes CVRANAL to analyze all files in the current directory whose name fits the template a*.cvr. Configuration options are read from cvrfred.fig, but with item 513 reset to 0. The output report is sent to cvrfred.out.

4.3 The CVRANAL Configuration File

The CVRANAL configuration file has the same general format as the SPAG and GXCHK configuration files (see Section 2.5, 3.3). The item numbers for CVRANAL are in the range 500-600, and all except one are integers. There is no overlap between CVRANAL, GXCHK and SPAG item numbers.

An example configuration file is shown below. The meaning of each data item is explained in detail in the next section. The values shown in this example are, in fact, the defaults which are used by CVRANAL if no configuration file is found.

Example CVRANAL Configuration File

 =========================== CVRANAL controls ==============================

501=0   0 - normal run                        1 - set usage counts to zero
502=0   0 - Unicode box drawing chars         1 - use only ASCII characters
503=200 number of lines/page in CVRANAL.OUT (if >=200 no pages within reports)
504=78  width of output reports

510=2   0=No check for unexercised code
        1=Report unexercised code to CVRANAL.OUT
        2=line number summary to CVRANAL.OUT
511=200 Number of hotspots to CVRANAL.OUT
512=7   Number of hotspots to screen
513=73  0=don't modify source code  >0  insert usage count at column n

550=          List (separated by ;) of directories containing source code

4.4 CVRANAL Configuration Data

4.5 Using CVRANAL

4.5.1 Source Code Changes

CVRANAL may be used as often as required, at any stage during the testing process. However if the source code is changed in any way, it is necessary to use SPAG to re-instrument the changed code before compiling and linking the revised test program. Once this is done, we recommend that you re-initialize all the coverage data (by running CVRANAL with item 501 set to 1), even for source files which have not been changed. If this is not done, the significance of the execution hot-spot data is questionable, as different parts of the code will be using different base points.

4.5.2 Execution Count Overflow

In the probe routines as supplied, execution counts are stored in a standard Fortran INTEGER array, and there is no guard against overflow. On most machines, this means that if a block of code is executed more than about 2147 million times, the execution count becomes very large and negative. In most cases, this is an acceptable compromize, as it is rare for execution counts to be so large. However, a version of the probe routines in which execution counts are double precision floating point numbers may easily be constructed by changing the type of the variable TCOunt from INTEGER to DOUBLE PRECISION in the probes source code file (probes.f90 in the plusFORT installation directory). This modified version may be slower, and uses more memory, but has an effectively unlimited capacity.

4.5.3 Troubleshooting

• If CVRANAL reports that all code blocks are untested, even in routines that you know must have been executed, then it is very likely that the test program did not finish normally. In this case you may also find that the timing reports have not been written to probes.log. The .cvr files are updated by the call to the routine (normally called PR$EXI) which SPAG inserts before every STOP statement, and before the END statement of the main PROGRAM. If the test program halts in some other way, for example because of a run-time error, or because the user interrupted execution, then the .cvr files will not be updated to reflect the usage in the current run. If in doubt, check the last modification time of probes.log and the the .cvr files

  This can also happen if the program is halted by a call to routine which has not been instrimented by SPAG. For example, many compilers provide a run-time routine (e.g EXIT) to halt execution. In such cases, it is necessary to hand-edit the instrumented code to insert a call to PR$EXI before the call to EXIT. Thus:-

        CALL BL$ENT(2158)
        CALL EXIT(16)
  

  would be changed to:-

        CALL BL$ENT(2158)
        CALL PR$EXI
        CALL EXIT(16)
  

• The probe routines use two Fortran I/O unit numbers, by default 67 and 68, for reading and writing coverage data and log files. If the calling program also uses those unit numbers, input and output will be garbled. To avoid this problem, simply change the values defined near the beginning of probes.f90

• The probe routines contain code which checks for error conditions, and may halt execution with one of the following messages, for example, if a very large program causes overflow of an internal array:-

        Profiler - Slot Overflow
        Profiler - File list too long
        Profiler - Total line limit exceeded
        Profiler - File buffer overrun
        Profiler - Unable to open coverage data file
        Profiler - Invalid traceback in SB$EXI
  

  The cure in the first four cases is to increase the dimensions (MAXSUB, MAXLIN and BUFSIZ) specified in probes.f90 and rebuild the instrumented program. Please note the comments in probes.f90 regarding possible values of MAXSUB. The fifth may indicate that a .cvr file has been moved or deleted, or has been locked by another program. The last message indicates a logical error, and should not happen unless the source files were incorrectly hand-edited.