Regression testing is used by software developers to ensure that program modifications have not negatively impacted the correctness of code. While regression testing has been successfully applied in many domains, programs such as web applications, XML processors, and compilers remain expensive to test because harmless program evolutions make the tests appear to fail: in our experiments 82% of test case output differences are false positives.We present an automated tool that measures syntactic differences in the tree-structured output of such programs to reduce the number of false positives in, and thus the cost of, regression testing. We model test case outputs that merit human inspection through a set of structural and domainspecific features. We evaluate the performance of our technique on over 20,000 test case output comparisons, and find that we are three times as accurate as a naive comparator.

Web-based applications are one of the most widely used types of software, and have become the backbone of many e-commerce and communications businesses. These applications are often mission-critical for many organizations, motivating their precise validation. Although regression testing has been widely used to gain confidence in the reliability of software by providing information about the quality of an application, it has suffered limited use in this domain due to the frequent nature of updates to websites and the difficulty of automatically comparing test case output. We present techniques to address these challenges in regression testing web-based applications. Without precise comparators, test cases that fail due to benign program evolutions must be manually inspected. Our approach harnesses the inherent similarities between unrelated web-based applications to provide fully automated solutions to reduce the number of such false positives, while simultaneously returning true faults. By applying a model derived from regression testing other programs, our approach can predict which test cases merit human inspection. Our method is 2.5 to 50 times as accurate as current industrial practice, but requires no user annotations.