Experimentation growth: Evolving trustworthy A/B testing capabilities in online software companies

Abstract

Companies need to know how much value their ideas deliver to customers. One of the most powerful ways to accurately measure this is by conducting online controlled experiments (OCEs). To run experiments, however, companies need to develop strong experimentation practices as well as align their organization and culture to experimentation. The main objective of this paper is to demonstrate how to run OCEs at large scale using the experience of companies that succeeded in scaling. Based on case study research at Microsoft, Booking.com, Skyscanner, and Intuit, we present our main contribution—The Experiment Growth Model. This four‐stage model addresses the seven critical aspects of experimentation and can help companies to transform their organizations into learning laboratories where new ideas can be tested with scientific accuracy. Ultimately, this should lead to better products and services.

Evaluating Personalization: The AB Testing Pitfalls Companies Might Not Be Aware of—A Spotlight on the Automotive Sector Websites

“…With it, declaring a winner version of the test or deciding if the hypothesis is validated can become a difficult task and the final conclusions of the test might be left to the personal interpretation, which is the opposite of what a web testing initiative should stand for Kohavi et al (2007). Moreover, the results are in line with the Experimentation Growth Model (Fabijan et al, 2018a). The companies with greater experience on AB testing report the use of stable metrics along their experiments, while companies with less experience report sets of evaluation metrics highly dependent on the specific experiment.…”

“…Typically, most of the publications came from big digital companies, such as Microsoft (Dmitriev et al, 2017), Google (Hohnhold et al, 2015), Facebook (Bakshy et al, 2014), Uber (Deb et al, 2018), or Netflix (Amatriain and Basilico, 2012;Su and Yohai, 2019). However, both small-to-medium companies and also big traditional companies are now adopting website experimentation initiatives (Olsson et al, 2017;Fabijan et al, 2018a). From the observation of some of those initiatives in companies of the automotive sector (commonly seen as traditional industrial companies), we identified and reported some critical pitfalls for the reliability of AB tests that were repeated with worrying regularity (Esteller-Cucala et al, 2019).…”

Data-Driven Software Development at Large Scale : from Ad-Hoc Data Collection to Trustworthy Experimentation

“…Product teams experiment with their design decisions, relevance modifications, infrastructure changes and other types of features in order to conduct a decision. In the early stages of experimentation maturity [9], [229], experimenters optimize their products on experiment specific metrics. Ideally, as companies mature, scorecards should clearly mark the subset of key metrics that the product organization agreed on being optimized for.…”

“…In previous research, we inductively derived the Experimentation Growth Model (EGM model) from analyzing the experience of growing experimentation activities in over a dozen Microsoft products an further detailed our work through case studies at Skyscanner, Booking, and Intuit [9], [229].…”

“…For example, building data pipeline is challenging, and conclusions can be entirely wrong when experiments interfere with each other, or when the randomization of users is biased by previous experiments [15], [16], [187]- [189]. In our previous research, we described the evolutionary stages of experimentation growth in detail through the Experimentation Growth Model (EGM model) [9], [229]. Our experience from applying A/B testing in various Microsoft products for the last 10 years, as well as the experience at other experimentation-driven companies that we collaborate with, has been that enabling and growing experimentation is a gradual process full of challenges and pitfalls [15], [16], [187].…”