|User Guide / Working with test results|
Comparing the results of two test runs is useful in tracking and debugging new and existing issues in the AUT.
The results of a given test run can be saved to a repository and serve as a snapshot of the state of the AUT (or a subset of the functions of the AUT) at that specific point in time. Progress in the development cycle can thereafter be measured by comparing a recent test run with an archived test run.
A test results record for a given automation run has a Summary tab, which summarizes various aspects of the test, and a Result Details tab, which displays a detailed (line-by-line) log of the test execution. The information detailed in the test results is helpful for pinpointing test problems, but searching test results line-by-line for clues is time-consuming, especially when comparing multiple test results.
Test summaries, on the other hand, are succinct, and comparing one test summary with another can be done relatively quickly. That said, test summaries do not reveal anything about the nature of bugs. In fact, they can be misleading as an indicator of progress. For example, a test run on Tuesday might result in 15 failures while a Wednesday run ends with only 7 failures. By reading the summary, it may be tempting to conclude that 8 bugs have been fixed; in fact, there is no assurance of that. It is possible that the original 15 bugs were all resolved and 8 new bugs have been introduced. Test summaries do not tell you the seriousness of bugs either. An overall reduction in the number of bugs is no guaranty that the number of critical bugs has decreased.
TestArchitect offers two methods of comparing test results: Compare Results and Compare to Baseline Results. In both result comparison methods, TestArchitect shows the listing of comparison checkpoints in the current test results alongside the corresponding checkpoints from previous tests. This allows for a clear side-by-side comparison, enabling you to identify which bugs have been resolved, which bugs have not yet been addressed, and which new bugs have been introduced.