Controlled experiments on the web: survey and practical guide

Abstract

The web provides an unprecedented opportunity to evaluate ideas quickly using controlled experiments, also called randomized experiments, A/B tests (and their generalizations), split tests, Control/Treatment tests, MultiVariable Tests (MVT) and parallel flights. Controlled experiments embody the best scientific design for establishing a causal relationship between changes and their influence on user-observable behavior. We provide a practical guide to conducting online experiments, where endusers can help guide the development of features. Our experience indicates that significant learning and return-on-investment (ROI) are seen when development teams listen to their customers, not to the Highest Paid Person's Opinion (HiPPO). We provide several examples of controlled experiments with surprising results. We review the important ingredients of running controlled experiments, and discuss their limitations (both technical and organizational). We focus on several areas that are critical to experimentation, including statistical power, sample size, and techniques for variance reduction. We describe common architectures for experimentation systems and analyze their advantages and disadvantages. We evaluate randomization and hashing techniques, which we show are not as simple in practice as is often assumed. Controlled 123Controlled experiments on the web 141 experiments typically generate large amounts of data, which can be analyzed using data mining techniques to gain deeper understanding of the factors influencing the outcome of interest, leading to new hypotheses and creating a virtuous cycle of improvements. Organizations that embrace controlled experiments with clear evaluation criteria can evolve their systems with automated optimizations and real-time analyses. Based on our extensive practical experience with multiple systems and organizations, we share key lessons that will help practitioners in running trustworthy controlled experiments.

Online controlled experiments at large scale

“…We have previously made the case that in the online world, agility and continuous availability of users makes MVTs less appealing [18]. Researchers at Google made similar observations [3].…”

“…Multiple papers and books have been written on how to run an online controlled experiment [18; 7; 19; 20] and we will not address that here; we follow the terminology of Controlled experiments on the web: survey and practical guide [18]. We build upon that work and share how to scale experimentation, i.e., how to run many experiments to accelerate innovation in product development.…”

“…Sessions per user, or repeat visits, is a much better factor in the OEC, and one that we use at Bing. Thinking of the drivers of lifetime value can lead to a strategically powerful OEC [18]. We cannot overemphasize the importance of coming up with a good OEC that the organization can align behind, but for this paper we will assume this has been done.…”

“…As a request is received from a browser, Bing's frontend servers assign each request to multiple flights running on a set of number lines. To ensure the assignment is consistent, a pseudo random hash of an anonymous user id is used [18]. The assignment happens as soon as the request is received and the frontend then passes each request's flight assignments as part of the requests sent to lower layers of the system.…”

Online Evaluation for Information Retrieval

“…These factors have significantly contributed to the rapid adoption of online evaluation techniques in these settings. In industry, online evaluation approaches such as AB tests (c.f., Section 2.4) and interleaved comparisons (Section 2.6) are now the state of the art for evaluating system effectiveness [Kohavi et al, 2009, Radlinski and Craswell, 2010, Bendersky et al, 2014.…”