An empirical evaluation and comparison of manual and automated test selection

Abstract

Regression test selection speeds up regression testing by rerunning only the tests that can be affected by the most recent code changes. Much progress has been made on research in automated test selection over the last three decades, but it has not translated into practical tools that are widely adopted. Therefore, developers either re-run all tests after each change or perform manual test selection. Re-running all tests is expensive, while manual test selection is tedious and error-prone. Despite such a big trade-off, no study assessed how developers perform manual test selection and compared it to automated test selection.This paper reports on our study of manual test selection in practice and our comparison of manual and automated test selection. We are the first to conduct a study that (1) analyzes data from manual test selection, collected in real time from 14 developers during a three-month study and (2) compares manual test selection with an automated state-of-the-research test-selection tool for 450 test sessions.Almost all developers in our study performed manual test selection, and they did so in mostly ad-hoc ways. Comparing manual and automated test selection, we found the two approaches to select different tests in each and every one of the 450 test sessions investigated. Manual selection chose more tests than automated selection 73% of the time (potentially wasting time) and chose fewer tests 27% of the time (potentially missing bugs). These results show the need for better automated test-selection techniques that integrate well with developers' programming environments.

Usage of Tests in an Open-Source Community

“…We now present our results and compare them to the three previous case studies of Beller et al [2015], Blondeau et al [2017], Gligoric et al [2014].…”

“…We have 20 participants. So, our study is closer to the third paper Gligoric et al [2014] which have 14 developers as participants. The second paper Beller et al [2015] have 48.…”

“…We could observe whether there is the same phenomenon in an open-source community. Blondeau et al [2017] also took inspiration from Gligoric et al [2014] and Beller et al [2015]. The results of these two other papers will be compared to ours in Section 4.…”

“…When a developer saves one of its modification, most of the time only one line is changed. Gligoric et al [2014] used the number of AST node differences between two versions of the code.…”

“…This research would cover well our need to characterize the testing behavior of developers. Third, Gligoric et al [2014] is a "Comparison of Manual and Automated Test Selection". As such, it seems to match well our second goal of verifying whether test selection can improve testing practice.…”