An approach and benchmark to detect behavioral changes of commits in continuous integration

Abstract

When a developer pushes a change to an application's codebase, a good practice is to have a test case specifying this behavioral change. Thanks to continuous integration (CI), the test is run on subsequent commits to check that they do no introduce a regression for that behavior.In this paper, we propose an approach that detects behavioral changes in commits. As input, it takes a program, its test suite, and a commit. Its output is a set of test methods that capture the behavioral difference between the pre-commit and postcommit versions of the program. We call our approach DCI (Detecting behavioral changes in CI). It works by generating variations of the existing test cases through (i) assertion amplification and (ii) a search-based exploration of the input space.We evaluate our approach on a curated set of 60 commits from 6 open source Java projects. To our knowledge, this is the first ever curated dataset of real-world behavioral changes. Our evaluation shows that DCI is able to generate test methods that detect behavioral changes. Our approach is fully automated and can be integrated into current development processes. The main limitations are that it targets unit tests B. Danglot Inria Lille -Nord Europe

Developer-centric test amplification

“…Because of the high computational cost of test generation, many tools have opted for integration into the continuous integration process (Arcuri et al 2016;Danglot et al 2020). This, however, leads to a long time distance between triggering the test generation and receiving results (Beller et al 2017), as well as the developers having to inspect the tools outside of their familiar development environment.…”

AmPyfier: Test amplification in Python

“…18,19 AMP add is a special case of AMP change , as its tools and techniques only focus on the changed parts of the project under test compared to its previous version (e.g., DCI). 20,21 On the other hand, AMP exec will modify the test suites dependencies like the OS file system, libraries, databases, remote services, or access APIs to GPS or Bluetooth (e.g., CAMP). 22,23 Finally, AMP mod techniques will try to make the test suite more precise by increasing the input exploration (e.g., TAUTOKO) 24 or regenerating the oracles (e.g., Orstra).…”

A Differential Testing Approach for Evaluating Abstract Syntax Tree Mapping Algorithms

“…Moreover, many studies used GumTree to analyze code patterns of changes such as bug-fixing changes [5], [13], [18], [21], [23], [29], logging changes [22] and changes to online code examples [34]. Also, prior work trained models based on the edit actions of changes that are calculated using GumTree [7], [15], [24], [31], [32]. Such models are used to recommend changes such as patches [32] and logging changes [22].…”