Building blocks for continuous experimentation

Abstract

Development of software-intensive products and services increasingly occurs by continuously deploying product or service increments, such as new features and enhancements, to customers. Product and service developers need to continuously find out what customers want by direct customer feedback and observation of usage behaviour, rather than indirectly through up-front business analyses. This paper examines the preconditions for setting up an experimentation system for continuous customer experiments. It describes the building blocks required for such a system. An initial model for continuous experimentation is analytically derived from prior work. The model is then matched against empirical case study findings from a startup company and adjusted. Building blocks for a continuous experimentation system and infrastructure are presented. A suitable experimentation system requires at least the ability to release minimum viable products or features with suitable instrumentation, design and manage experiment plans, link experiment results with a product roadmap, and manage a flexible business strategy. The main challenges are proper and rapid design of experiments, advanced instrumentation of software to collect, analyse, and store relevant data, and the integration of experiment results in both the product development cycle and the software development process.

Early Value Argumentation and Prediction: An Iterative Approach to Quantifying Feature Value

“…The novel development approach as pictured in the QCD model captures, and expand further, on ideas that have been previously published [3], [12], [13]. In the QCD model quantitative, as well as qualitative feedback techniques are recognized.…”

Hitting the Target: Practices for Moving Toward Innovation Experiment Systems

“…The adoption of these practices reflects an evolution in which companies move beyond agile practices towards R&D practices characterised by short release cycles, frequent customer validation and fully automated testing and deployment practices. Although the same agile R&D principles apply, moving beyond agile practices means: a) integrating business strategy planning, operations and other corporate functions into shorter development and release cycles [4], [15]; b) utilising automated testing practices that allow for frequent builds [12] and c) implementing continuous experimentation and innovation with customers [2,3,4] to better understand real customer needs. The specific aspects involved in going beyond agile as well as more holistic views of agility have been discussed in recent SE studies [15,16] and especially in the context of lean software development [17].…”

“…In step E, companies adopt data collection mechanisms to continuously learn about customer behaviour and product use. Feature experiments are run on a continuous basis and the collected data steer the R&D organisation [2,3]. Rather than being specified by the product management in the early phase of development, requirements evolve based on data collected from real-time customer use.…”

“…To solve this problem, agile methods [1] offer a set of practices that allow for shorter development cycles and more frequent interaction with customers. In conjunction with agile methodologies, approaches, such as innovation experiment systems (IES) [2] and continuous experimentation [3], emphasise data collection practices and continuous validation with customers in order to improve R&D accuracy and customer responsiveness. However, while these approaches are attractive to companies in the software industry, they require an evolution of the company's current ways of working.…”

Time to Say 'Good Bye': Feature Lifecycle

“…In this way, value is assumed to be generated to customers and users continuously with every update and deployment [2]. However, although companies are adopting techniques to validate the value of the features during development, shortly after and while they are being deployed [3], [4], [5], [6], [7], very little is known on how to track feature value over its lifetime and what actions to perform when a feature no longer adds value to the product.…”