Improving the Sensitivity of Online Controlled Experiments

Xie, Hujun; Aurisset, Juliette

doi:10.1145/2939672.2939733

Cited by 87 publications

(38 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although the Delta method can help tackle big data problems, it does not replace the need for rigorous experimental designs and probabilistic modeling. For example, "optimal" choices for cluster configurations, randomization mechanisms and data transformations are known to increase the sensitivities of metrics [8,32,49]. We leave them as future research directions.…”

Section: Discussionmentioning

confidence: 99%

Applying the Delta Method in Metric Analytics

Deng

Knoblich

2018

Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

During the last decade, the information technology industry has adopted a data-driven culture, relying on online metrics to measure and monitor business performance. Under the setting of big data, the majority of such metrics approximately follow normal distributions, opening up potential opportunities to model them directly without extra model assumptions and solve big data problems via closed-form formulas using distributed algorithms at a fraction of the cost of simulation-based procedures like bootstrap. However, certain attributes of the metrics, such as their corresponding data generating processes and aggregation levels, pose numerous challenges for constructing trustworthy estimation and inference procedures. Motivated by four real-life examples in metric development and analytics for large-scale A/B testing, we provide a practical guide to applying the Delta method, one of the most important tools from the classic statistics literature, to address the aforementioned challenges. We emphasize the central role of the Delta method in metric analytics by highlighting both its classic and novel applications.

show abstract

Section: Discussionmentioning

confidence: 99%

Applying the Delta Method in Metric Analytics

Deng

Knoblich

2018

Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

show abstract

“…Another approach is to apply adjustment during the analysis phase, using post-stratification or CUPED [40]. Netflix [41] has a nice comparison paper on some of these approaches.…”

Section: Similarities and Dissimilarities With Medical Rctsmentioning

confidence: 99%

Online randomized controlled experiments at scale: lessons and extensions to medicine

Kohavi

Tang

et al. 2020

Trials

View full text Add to dashboard Cite

Background: Many technology companies, including Airbnb, Amazon, Booking.com, eBay, Facebook, Google, LinkedIn, Lyft, Microsoft, Netflix, Twitter, Uber, and Yahoo!/Oath, run online randomized controlled experiments at scale, namely hundreds of concurrent controlled experiments on millions of users each, commonly referred to as A/ B tests. Originally derived from the same statistical roots, randomized controlled trials (RCTs) in medicine are now criticized for being expensive and difficult, while in technology, the marginal cost of such experiments is approaching zero and the value for data-driven decision-making is broadly recognized. Methods and results: This is an overview of key scaling lessons learned in the technology field. They include (1) a focus on metrics, an overall evaluation criterion and thousands of metrics for insights and debugging, automatically computed for every experiment; (2) quick release cycles with automated ramp-up and shutdown that afford agile and safe experimentation, leading to consistent incremental progress over time; and (3) a culture of 'test everything' because most ideas fail and tiny changes sometimes show surprising outcomes worth millions of dollars annually. Technological advances, online interactions, and the availability of large-scale data allowed technology companies to take the science of RCTs and use them as online randomized controlled experiments at large scale with hundreds of such concurrent experiments running on any given day on a wide range of software products, be they web sites, mobile applications, or desktop applications. Rather than hindering innovation, these experiments enabled accelerated innovation with clear improvements to key metrics, including user experience and revenue. As healthcare increases interactions with patients utilizing these modern channels of web sites and digital health applications, many of the lessons apply. The most innovative technological field has recognized that systematic series of randomized trials with numerous failures of the most promising ideas leads to sustainable improvement. Conclusion: While there are many differences between technology and medicine, it is worth considering whether and how similar designs can be applied via simple RCTs that focus on healthcare decision-making or service delivery. Changessmall and largeshould undergo continuous and repeated evaluations in randomized trials and learning from their results will enable accelerated healthcare improvements.

show abstract

“…David and Nagaraja [26] have provided a formula to estimate the covariance between Y (r) and Y (s) for any r < s ≤ N: 5 To obtain the covariance between X I(r) and X I(s) for any r, s ≤ N , we again refer to [28] (Eq. 2.3d):…”

Section: Var(v 1 )mentioning

confidence: 99%

“…The value of E&M is currently best reflected in the success of major organizations that have adopted and advocated for them in the past decade. A large number of major technology companies report having mature infrastructure for online-controlled experiments (OCEs, e.g., Google [1], Linkedin [2], and Microsoft [3]) and/or are heavily investing in state-of-the-art techniques (e.g., Airbnb [4], Netflix [5], and Yandex [6]). Amazon [7] and Facebook [8] have also reported the use of various causal inference techniques to measure the incrementality of advertising campaigns.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

What is the Value of Experimentation and Measurement?

Liu

Chamberlain²,

McCoy

2020

Data Sci. Eng.

View full text Add to dashboard Cite

Experimentation and Measurement (E&M) capabilities allow organizations to accurately assess the impact of new propositions and to experiment with many variants of existing products. However, until now, the question of measuring the measurer, or valuing the contribution of an E&M capability to organizational success has not been addressed. We tackle this problem by analyzing how, by decreasing estimation uncertainty, E&M platforms allow for better prioritization. We quantify this benefit in terms of expected relative improvement in the performance of all new propositions and provide guidance for how much an E&M capability is worth and when organizations should invest in one.

show abstract

Improving the Sensitivity of Online Controlled Experiments

Cited by 87 publications

References 8 publications

Applying the Delta Method in Metric Analytics

Applying the Delta Method in Metric Analytics

Online randomized controlled experiments at scale: lessons and extensions to medicine

What is the Value of Experimentation and Measurement?

Contact Info

Product

Resources

About