The Effects of Continuous Integration on Software Development: a Systematic Literature Review

Abstract

Context: Continuous integration (CI) is a software engineering technique that proclaims a set of frequent activities to assure the health of the software product. Researchers and practitioners mention several benefits related to CI. However, no systematic study surveys state of the art regarding such benefits or cons. Objective: This study aims to identify and interpret empirical evidence regarding how CI impacts software development. Method: Through a Systematic Literature Review, we search for studies in six digital libraries. Starting from 479 studies, we select 101 empirical studies that evaluate CI for any software development activity (e.g., testing). We thoroughly read and extract information regarding (i) CI environment, (ii) findings related to effects of CI, and (iii) the employed methodology. We apply a thematic synthesis to group and summarize the findings. Results: Existing research has explored the positive effects of CI, such as better cooperation, or negative effects, such as adding technical and process challenges. From our thematic synthesis, we identify six themes: development activities, software process, quality assurance, integration patterns, issues & defects, and build patterns. Conclusions: Empirical research in CI has been increasing over recent years. We found that much of the existing research reveals that CI brings positive effects to the software development phenomena. However, CI may also

Investigating the Impact of Continuous Integration Practices on the Productivity and Quality of Open-Source Projects

“…We were inspired by Felidré et al's work on the CI Theater phenomenon [10], which revealed that many projects do not follow CI practices. Therefore, instead of considering the usage of a CI server as the sole criterion to assume that projects use CI (which is a problem also highlighted by Soares et al 's work [25]), we investigated the potential impact of CI on quality and productivity outcomes by means of 5 CI subpractices.…”

“…Existing research still lacks a well-defined set of criteria to analyze the potential impact of CI on software development [25]. For instance, several studies consider the use of a CI service (e.g., Travis-CI) as the sole criterion to indicate whether projects use CI or not [25].…”

“…Existing research still lacks a well-defined set of criteria to analyze the potential impact of CI on software development [25]. For instance, several studies consider the use of a CI service (e.g., Travis-CI) as the sole criterion to indicate whether projects use CI or not [25]. However, Duvall et al [9] and Fowler [11] have defined a set of CI sub-practices that, combined together, can be used to determine whether a project uses CI or not.…”

“…If there were no builds, the value would be 0. • Build Health [25] is a unit interval representing the rate of build failures across days. If there were build failures every day, the value would be 0. if there were no build failures, the value would be 1.…”

“…Over the last years, researchers have empirically studied CI from different perspectives [25] [20] [32] [18] [24]. Recent research works have investigated the impact of CI on productivity and quality of software projects [30] [2].…”