Continuous Integration and Quality Assurance: a case study of two open source projects

Abstract

A decentralized variant of continuous integration can be defined in terms of two fundamental rules: (1) Developers' access to add contributions to the development version at any time, and (2) developers' obligation to integrate their own contributions properly. Decentralized, continuous integration may adapt well to organizations where developers work relatively independently, as in many open source projects. The approach raises the issue of how these organizations can exercise central control, as attaining the benefits of continuous integration requires that contributions are useful and satisfy the project's definition of successful integration. We have investigated the use of continuous integration in FreeBSD and Mozilla. Our account of quality assurance activities in the two open source projects distinguishes between Mintzberg's three complementary forms of central control: Standardization and control of work output, work processes, and worker skills. Our study indicates that two major challenges face projects using decentralized, continuous integration: (1) To balance the access to add contributions against the need to stabilize and mature the software prior to a release, and (2) to consider the developers' limited time and resources when interpreting their obligation to integrate their changes properly.

Challenges When Adopting Continuous Integration: A Case Study

“…Deciding when to integrate changes is also delegated to the developer. This is an advantage when compared to the more plan-driven work assignment process of traditional projects [7]. Breaking down tasks into smaller pieces is also prevalent in open source.…”

“…Holck and Jorgensen [7] studied the use of a decentralized CI process in the open source community (FreeBSD and Mozilla) where developers are often distributed. The authors found that developers working on open source projects are often free to pick any tasks or bugs they want to work on.…”

The impact of continuous integration on other software development practices: A large-scale empirical study

“…The earlier studies [36], [37] interpret CI as distributed development and obligation to integrate one's own contributions. Under this definition no commit size reduction has been observed in 5,122 Ohloh projects [37].…”

CiCUTS: Combining System Execution Modeling Tools with Continuous Integration Environments

“…Prior work has also explored the benefits of using continuous integration environments, such as automating the build process [4], only releasing modules after they pass all automated test cases [23], and reducing integration risk by finding errors earlier in the lifecycle [12]. Our work on CiCUTS also shows the benefits of using continuous integration environments to automate key aspects of the testing process.…”

“…In practice, however, these drawbacks are not problematic because agreeing on an intermediate format is straightforward. Moreover, continuous integration environments are used in industrial development [4,12] and support extension for domain-specific needs, such as evaluating the success of results generated by domain-specific extensions.…”