Work practices and challenges in continuous integration: A survey with Travis CI users

Abstract

Continuous integration (CI) is a software development practice that has been gaining increasing popularity in the last few years. However, we still miss a collection of experiences regarding how software developers perceive the idea of CI, in terms of its fundamental concepts, the reasons that motivate the adoption of this practice, the reasons for build breakage, and the benefits and problems related to CI. To shed light on this direction, we conducted a user survey with 158 CI users. Through a mostly qualitative investigation, we produce a list of findings that are not always obvious. For instance, we observed that (1) developers are not sure whether a job failure represents a failure or not; (2) inadequate testing is the most common technical reason related to build breakage, whereas lack of time plays a role on the social reasons; and (3) although some respondents reported that CI systems increase the confidence that the code is in a known state, some respondents also reported that there is a false sense of confidence when blindly trusting tests. This empirical study is particularly relevant to those interested in better understanding and fostering CI practices either in an open-source or industrial setting.

Continuous Integration Theater

“…Although the culture of continuously integrating changes dates from the 70s [2], CI practices has gained momentum only in the last 10 years, being more widely discussed, employed, and researched. Consequently, CI is nowadays one of the pillars of the software engineering practice, not only in commercial projects, but also in open source projects [3], [4].…”

“…In this paper, we investigate a set of CI bad practices. These bad practices are related to the use of CI (1) with infrequent commits on the master branch (i.e., delaying integration), (2) in a software project with poor test coverage (i.e., missing eventual bugs), (3) with builds that remain broken for long periods for time (i.e., blocking new features), and (4) with builds with considerably long durations (i.e., limiting the rapid feedback). These bad practices constitute what is known as the Although these bad practices are commonly discussed in the grey literature, little research has been devoted to shed some light on the existence of projects performing the CI Theater.…”

Traceability Establishment and Visualization of Software Artefacts in DevOps Practice: A Survey

“…DevOps environments associate five main principles: continuous planning, continuous integration, continuous delivery, continuous testing and continuous deployment [6]. The CI process integrates work frequently that leads to multiple integrations per day and deploys effectively [17] [18]. Generally, CI verifies the integration using build automation by early detection of integration errors.…”

The Effects of Continuous Integration on Software Development: a Systematic Literature Review

“…Given the increasing popularity of CI along with its claimed benefits, there have been substantial research efforts related to CI. Researchers have investigated CI practices [29,38,43], environments & tools [22,37,44], potential benefits, [2,15,35], potential problems [8,20,30,39], and even new practices [27,32,42] in diverse settings. Given these valuable and recent research contributions, our community is in need of a clear map of the empirical benefits or cons of adopting CI.…”