The Evolution of Continuous Experimentation in Software Product Development: From Data to a Data-Driven Organization at Scale

Abstract

Abstract-Software development companies are increasingly aiming to become data-driven by trying to continuously experiment with the products used by their customers. Although familiar with the competitive edge that the A/B testing technology delivers, they seldom succeed in evolving and adopting the methodology. In this paper, and based on an exhaustive and collaborative case study research in a large software-intense company with highly developed experimentation culture, we present the evolution process of moving from ad-hoc customer data analysis towards continuous controlled experimentation at scale. Our main contribution is the "Experimentation Evolution Model" in which we detail three phases of evolution: technical, organizational and business evolution. With our contribution, we aim to provide guidance to practitioners on how to develop and scale continuous experimentation in software organizations with the purpose of becoming data-driven at scale. A/B

Experimentation growth: Evolving trustworthy A/B testing capabilities in online software companies

“…In the first phase, the first two authors of this paper conducted an in‐depth single case study at Microsoft. The outcome of the first phase was the Experimentation Evolution Model . The second phase of this research consisted of a follow‐up multiple case study, where we worked with Microsoft and with representatives from three additional large‐scale online software companies to further detail and generalize our initial findings from phase 1.…”

“…Systematically conducting trustworthy experiments at large‐scale requires a transformation that is far from intuitive. Conducting many OCEs annually requires technical capabilities that need to be developed, organizational changes that ingrain experimentation with companies' culture and practices, and alignments of business objectives with experiment metrics . The main research question that we strive to answer in this work is “how do large‐scale online software companies grow their experimentation capabilities?” The main contribution of this paper, and the answer to our research question, is The Experimentation Growth Model.…”

Search-Based Scheduling of Experiments in Continuous Deployment

“…The core ingredients for all these types of experiments are the same. Once the responsible developer or analyst decides to launch an experiment, they need to have an understanding of (1) what to measure, i.e., the overall evaluation criterion (OEC) [16], [9], (2) how many data points to collect for being able to statistically reason about the OEC and thus the experiment's outcome (i.e., sample size), (3) which users to conduct the experiment on (e.g., different user groups, regions), and (4) when to run the experiment.…”

“…However, setting up such an experimentation infrastructure is not a straightforward task, as demonstrated by experience reports of, e.g., Kevic et al [1], Xu et al [8], and Fabijan et al [9]. An essential requirement for successful experimentation is to collect enough data to draw valid statistical conclusions (cf.…”

Continuous experimentation for software developers

“…Those include reports of Microsoft [10,11] and Google [20]. Similarly, Fabijan et al [5] derived a model detailing technical, organizational, and business evolution to provide a guidance towards data-driven experimentation based on an investigation of the evolution of the experimentation process at Microsoft.…”