KATCH: high-coverage testing of software patches

Abstract

One of the distinguishing characteristics of software systems is that they evolve: new patches are committed to software repositories and new versions are released to users on a continuous basis. Unfortunately, many of these changes bring unexpected bugs that break the stability of the system or affect its security. In this paper, we address this problem using a technique for automatically testing code patches. Our technique combines symbolic execution with several novel heuristics based on static and dynamic program analysis which allow it to quickly reach the code of the patch.We have implemented our approach in a tool called katch, which we have applied to all the patches written in a combined period of approximately six years for nineteen mature programs from the popular GNU diffutils, GNU binutils and GNU findutils utility suites, which are shipped with virtually all UNIX-based distributions. Our results show that katch can automatically synthesise inputs that significantly increase the patch coverage achieved by the existing manual test suites, and find bugs at the moment they are introduced.

Covrig: a framework for the analysis of code, test, and coverage evolution in real software

“…Our prior work on testing software patches [23] reported the aggregate patch coverage achieved by regression test suites, but the focus was on evaluating the patch testing technique proposed.…”

Shadow of a doubt

“…executes at least one patch statement-if such an input does not exist in the program's test suite, it could be generated using previous techniques such as KATCH [20].…”

“…At the very minimum, each line of code affected by the patch should be covered by at least one test case. While this level of testing is still far from being achieved in practice [21], automatic techniques for enabling high-coverage patch testing are becoming more and more successful [1,17,20,29,31]. Many of these techniques are based on dynamic symbolic execution [7], a program analysis technique that provides the ability to generate inputs that form high-coverage test suites.…”

Shadow symbolic execution for better testing of evolving software

“…Unfortunately, often many of these changes themselves give rise to a significant amount of program bugs: new code is by definition little tested in the field, and often introduces bugs that affect both old and new functionality [10,18]. Even relatively simple, self-contained patches which are meant to fix specific bugs often introduce new errors [10,28].…”

“…For example, KATCH [18] uses a simple textual differencing algorithm to find out what statements have been added or changed, while DiSE [22] employs static program slicing to determine the statements affected by the patch. While this can lead to significant savings, static analysis of the program differences is often imprecise, and can miss important pruning and prioritisation opportunities, particularly those which exploit dynamic value information.…”