The impact of test case summaries on bug fixing performance: An empirical investigation

Abstract

Automated test generation tools have been widely investigated with the goal of reducing the cost of testing activities. However, generated tests have been shown not to help developers in detecting and finding more bugs even though they reach higher structural coverage compared to manual testing. The main reason is that generated tests are difficult to understand and maintain. Our paper proposes an approach, coined TestScribe, which automatically generates test case summaries of the portion of code exercised by each individual test, thereby improving understandability. We argue that this approach can complement the current techniques around automated unit test generation or searchbased techniques designed to generate a possibly minimal set of test cases. In evaluating our approach we found that (1) developers find twice as many bugs, and (2) ABSTRACTAutomated test generation tools have been widely investigated with the goal of reducing the cost of testing activities. However, generated tests have been shown not to help developers in detecting and finding more bugs even though they reach higher structural coverage compared to manual testing. The main reason is that generated tests are difficult to understand and maintain. Our paper proposes an approach, coined TestDescriber, which automatically generates test case summaries of the portion of code exercised by each individual test, thereby improving understandability. We argue that this approach can complement the current techniques around automated unit test generation or searchbased techniques designed to generate a possibly minimal set of test cases. In evaluating our approach we found that (1) developers find twice as many bugs, and (2) test case summaries significantly improve the comprehensibility of test cases, which is considered particularly useful by developers.

Generating unit tests with descriptive names or: would you name your children thing1 and thing2?

“…Method names are not the only identiers involved in unit tests; as future work we will investigate how generating useful variable names inuences test understanding and maintainability. It is also conceivable to extend our technique to synthesize more elaborate explanations or summaries [33,45].…”

“…There are, however, alternative approaches that aim to achieve the same. Natural language test summaries [33] or test documentation [27] can help to make tests understandable [33]. Tests can be made easier to understand by simplifying them, for example by reducing the number of statements [25,26], by reducing the number of assertions [17], or by splitting tests to one for each assertion [43].…”

Search-Based Crash Reproduction and Its Impact on Debugging

“…Recently, some research work in software testing and debugging started involving user evaluations include the following: [62], [65], [20], [33], [67], [34], and [61]. Parnin and Orso [62] performed a preliminary study with 34 developers to investigate whether and to what extent using an automated debugging approach may aid developers in their debugging tasks.…”

“…To improve the comprehensibility of test cases which in turn could improve the number of faults found by developers, Panichella et al [61] proposed TestDescriber which automatically generates summaries of the portions of the code that is exercised by individual test cases. To assess the impact of their approach, Panichella et al [61] performed a controlled experiment with 33 human participants comprising of professional developers, senior researchers, and students. The results of their study show that using TestDescriber, (i) developers find twice as many bugs, and (ii) test case summaries improve the comprehensibility of test cases which were considered useful by developers.…”

A Systematic Comparison of Search Algorithms for Topic Modelling—A Study on Duplicate Bug Report Identification

“…Topic model techniques have been widely used in software engineering (SE) literature to extract textual information from software artifacts. Textual information is often used support software engineers to semi-automated various tasks, such as traceability link retrieval [2], identify bug report duplicates [27], automated summary generator [34,30], source code labeling [11], and bug localization [22]. Latent Dirichlet Allocation (LDA) is a topic model techniques, which has received much attention in the SE literature due to its ability to extract topics (cluster or relevant words) from software documents.…”