Which Commits Can Be CI Skipped?

Abstract

Continuous Integration (CI) frameworks such as Travis CI, automatically build and run tests whenever a new commit is submitted/pushed. Although there are many advantages in using CI, e.g., speeding up the release cycle and automating the test execution process, it has been noted that the CI process can take a very long time to complete. One of the possible reasons for such delays is the fact that some commits (e.g., changes to readme files) unnecessarily kick off the CI process. Therefore, the goal of this paper is to automate the process of determining which commits can be CI skipped. We start by examining the commits of 58 Java projects and identify commits that were explicitly CI skipped by developers. Based on the manual investigation of 1,813 explicitly CI skipped commits, we first devise an initial model of a CI skipped commit and use this model to propose a rule-based technique that automatically identifies commits that should be CI skipped. To evaluate the rule-based technique, we perform a study on unseen datasets extracted from ten projects and show that the devised rule-based technique is able to detect and label CI skip commits, achieving Areas Under the Curve (AUC) values between 0.56 and 0.98 (average of 0.73). Additionally, we show that, on average, our technique can reduce the number of commits that need to trigger the CI process by 18.16%. We also qualitatively triangulated our analysis on the importance of skipping the CI process through a survey with 40 developers. The survey results showed that 75% of the surveyed developers consider it to be nice, important or very important to have a technique that automatically flags CI skip commits. To operationalize our technique, we develop a publicly available prototype tool, called CI-SKIPPER, that can be integrated with any git repository and automatically mark commits that can be CI skipped.

What Helped, and what did not? An Evaluation of the Strategies to Improve Continuous Integration

“…2) Build-level granularity: Build-selection techniques [1], [2], [20], [27], [42] aim at automatically detect and label commits and builds that can be CI skipped. Some approaches [20], [27], [42] try to detect failing builds and skip those passing builds to achieve cost-saving.…”

“…Some approaches [20], [27], [42] try to detect failing builds and skip those passing builds to achieve cost-saving. Others [1], [2] aim at identifying commits that can be CI skipped. Figure 1 illustrates how they work in the simulation timeline.…”

“…To obtain these information, we built scripts to download the raw build logs from Travis and parse them to extract all of the information about test executions, such as test name, duration and outcome. Some techniques, e.g., [36], [2], require additional information that TravisTorrent does not provide for builds, such as the content of commit messages, changed source lines and changed file names. For that, we also mined additional information about commits in the projects' code repositories through Github.…”

“…We follow this approach for all the techniques based on machine learning, e.g., [36]. For techniques that do not require training, e.g., [2 ], we simply execute them over the same last 10 folds. For techniques that train on data from other projects, i.e., for cross-project technique variants, we also executed them over the same last-1 0 -fold timeline -and we divided them into 1 0 project folds to do cross-project cross-validation, i.e., for each project, the technique is trained on 90 other projects and tested on its last 1 0 fold data.…”

“…Computational-cost-reduction techniques aim to observe failures only -by selectively executing failing builds only, saving the cost of executing passing ones. They also may operate at two different levels of granularity, selecting: test executions e.g., [36], or build executions e.g., [2 ].…”