Engineering for a science-centric experimentation platform

Abstract

Netflix is an internet entertainment service that routinely employs experimentation to guide strategy around product innovations. As Netflix grew, it had the opportunity to explore increasingly specialized improvements to its service, which generated demand for deeper analyses supported by richer metrics and powered by more diverse statistical methodologies. To facilitate this, and more fully harness the skill sets of both engineering and data science, Netflix engineers created a science-centric experimentation platform that leverages the expertise of scientists from a wide range of backgrounds working on data science tasks by allowing them to make direct code contributions in the languages used by them (Python and R). Moreover, the same code that runs in production is able to be run locally, making it straightforward to explore and graduate both metrics and causal inference methodologies directly into production services.In this paper, we provide two main contributions. Firstly, we report on the architecture of this platform, with a special emphasis on its novel aspects: how it supports science-centric end-to-end workflows without compromising engineering requirements. Secondly, we describe its approach to causal inference, which leverages the potential outcomes conceptual framework to provide a unified abstraction layer for arbitrary statistical models and methodologies.

Representation-Aware Experimentation: Group Inequality Analysis for A/B Testing and Alerting

“…It allows us to understand whether changing a ranking algorithm improves search results, sending a notification increases app visits, or a bug fix that should do nothing has any surprising results. When possible, A/B testing is the gold standard for understanding treatment effects, and therefore companies are increasingly using A/B tests to make decisions (Yin and Hong, 2019;Diamantopoulos et al, 2020;Karrer et al, 2021). LinkedIn currently runs hundreds of A/B tests per day, analyzing the impact of both major and minor adjustments, and always attempting to make data-driven decisions built on insights from experiments (Xu et al, 2015).…”

Characterizing Interference Heterogeneity and Improving Estimation for Experiments in Networks

“…Controlled experiments, also known as "A/B testing," continue to serve as the cornerstone for making strategic decisions in business, including new product launches, marketing campaigns, and algorithm updates (Bakshy et al 2014, Kohavi et al 2020, Diamantopoulos et al 2020, Bojinov and Gupta 2022, Koning et al 2022. Through the random assignment of treatment or control groups, A/B testing facilitates the evaluation of causal, rather than merely correlational, impacts of a product intervention on business outcomes.…”

You Only Compress Once: Optimal Data Compression for Estimating Linear Models

“…First, it must be able to scale both to large sample sizes, which can be as large as hundreds of millions of observations, and to many features, sometimes in the thousands. Second, it should be reproducible and extensible such that software engineers and researchers can interact with, iterate on, and subsequently contribute to it (Diamantopoulos et al, 2020).…”

“…Addressing the second challenge, Netflix described an inclusive XP that makes use of single-machine computation for modeling, allowing it to be more interactive and consistent with the way researchers iterate (Diamantopoulos et al, 2020). As a result, researchers can reproduce analyses from the XP, iterate, follow up, and debug using Python and R, and then contribute improvements to statistical methodology back to the engineering systems.…”