What Helped, and what did not? An Evaluation of the Strategies to Improve Continuous Integration

Abstract

Continuous integration (CI) is a widely used practice in modern software engineering. Unfortunately, it is also an expensive practice -Google and Mozilla estimate their CI systems in millions of dollars. There are a number of techniques and tools designed to or having the potential to save the cost of CI or expand its benefit -reducing time to feedback. However, their benefits in some dimensions may also result in drawbacks in others. They may also be beneficial in other scenarios where they are not designed to help. In this paper, we perform the first exhaustive comparison of techniques to improve CI, evaluating 14 variants of 10 techniques using selection and prioritization strategies on build and test granularity. We evaluate their strengths and weaknesses with 10 different cost and time-tofeedback saving metrics on 100 real-world projects. We analyze the results of all techniques to understand the design decisions that helped different dimensions of benefit. We also synthesized those results to lay out a series of recommendations for the development of future research techniques to advance this area.

Semantic‐aware two‐phase test case prioritization for continuous integration

“…There are many proposed works on reducing the high costs utilized in the CI environment. Najafi's team and Jin's team conducted experiments to compare and analyse the performance of test case selection and prioritization methods in actual CI scenarios [4,5] and put forward the challenges during practical application of CI. Based on the challenges, some approaches have been proposed to prioritize or select the builds that are waiting to be executed due to the time and resource constraints in CI environments [27][28][29].…”

“…To address the difficulties in typical CI scenarios, many research works focused on test optimization: Najafi's team and Jin's team conducted experiments to compare and analyse the performance of test case selection and prioritization methods [4,5] and put forward the challenges in the practical CI testing scenario. Test case prioritization (TCP) technique plays an important role in reducing CI testing costs in the environment, which establishes a test case execution sequence to detect more early faults under limited time and resource constraints.…”

Comparing Modern Build Automation Tools for an Insurance Company

“…Meyer gives in [21] an overview on CI and its tools', which is helpful for a general look at the topic. More strategically oriented, Jin and Servant discuss in [22] strategies to improve CI.…”

GLITCH: Automated Polyglot Security Smell Detection in Infrastructure as Code

“…The following could bring added value: integration with continuous integration (CI) processes (e.g., GitHub actions), integration with popular IDEs, interactive reports (e.g., highlight vulnerable code), and explainable warnings. Since GLITCH is much faster than other state-of-the-art tools for analyzing Chef and Puppet scripts, it becomes more appealing to integrate GLITCH as part of a CI workflow [8].…”