(Adapted from article published in Research Disclosure number 465 (January 2003), page 171.) (The tool referred to is available in the VM Download Library.)

Script-driven test tool with variable hierarchical output in HTML format

Disclosed is a method for automatically generating hierarchically layered results of software tests. In viewing results of a large collection of automated tests, such as from a nightly test run, it is desirable to be able to see results at multiple levels. One needs to see what groups of tests failed, what individual sub-groups failed within major groupings, what individual tests failed, and even where in an individual test the failure occurred. This disclosure shows how to do this for script-based tests; the same method applies to displaying results that are not script based as long as each test definitely passed or failed.

Related test scripts are combined in a file. Some number of these files are grouped together to form a collection of tests known as a BUCKET. A BUCKET is embodied as a file containing a list of names of test scripts and/or names of BUCKET files. A hierarchy of BUCKETs can exist (see figure 1). Variable depth for the hierarchy is allowed. A test file or BUCKET included by more than one higher-order BUCKET is run only once. Recursive BUCKETs causing an infinite loop are not allowed.

The test driving program follows the BUCKET hierarchy and runs the test scripts in depth first search order, storing results in HTML-formatted files. The test file HTML output begins with one line for each test case script within the test file. Each of these lines has a colored (or other figurative) pass/fail indicator and an HTML link (using the # construct) to the location of the particular test's results within the same file. For BUCKET files also, the test driving program builds an HTML results file. The results file for a BUCKET includes one line for each sub-BUCKET or test file. The line is marked with color or some symbol to indicate pass or failure of the sub-BUCKET or test file. Because tests are run in depth first search order, complete results are known when it comes time to write each HTML file. Without any re-writing, this makes possible adding the colors (or zig-zags if display is for a color-blind person). A typical rendering: each error red. Green might indicate tests that were successful.

Within an HTML results file for a BUCKET are lines listing lower level BUCKETs or test files. Each of these lines is a hyperlink to the corresponding HTML file of the test or BUCKET referred to by that line. A single failure at the lower level causes the line to be colored to indicate failure. The failure count can also be listed. The test driving program thus builds a hierarchy of HTML results files corresponding to the hierarchy defined by the BUCKET files (see figure 2). For any previously-run tests or BUCKETs that are detected, a previously-built line in some BUCKET's HTML results file will exist and can be duplicated, including the hyperlink, into the current BUCKET results file. Upon completion, a user can view the highest level HTML file and click down as desired to see details.

The information provided, and views expressed on this site are my own and do not represent the IBM Corporation.

Return to Tim Greer's home page.