Dealing with Ratio Metrics in A/B Testing at the Presence of Intra-user Correlation and Segments

Nie, Keyu; Kong, Yinfei; Yuan, Ted Tao; Burke, Pauline Berry

doi:10.1007/978-3-030-62008-0_39

Cited by 5 publications

(2 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They are critical for the platform to successfully execute thousands of high quality and trustworthy experiments to promote the experimentation culture among product developers. In the future, we plan to implement other methods to continuously monitor experiment pre-existing bias in decision metrics, e.g., accumulative and ratio metrics [13].…”

Section: Summary and Future Workmentioning

confidence: 99%

Ensure A/B Test Quality at Scale with Automated Randomization Validation and Sample Ratio Mismatch Detection

Nie,

Zhang,

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

eBay's experimentation platform runs hundreds of A/B tests on any given day. The platform integrates with the tracking infrastructure and customer experience servers, provides the sampling service for experiments, and has the responsibility to monitor the progress of each A/B test. There are many challenges especially when it is required to ensure experiment quality at the large scale. We discuss two automated test quality monitoring processes and methodologies, namely randomization validation using population stability index (PSI) and sample ratio mismatch (a.k.a. sample delta) detection using sequential analysis. The automated processes assist the experimentation platform to run high quality and trustworthy tests not only effectively on a large scale, but also efficiently by minimizing false positive monitoring alarms to experimenters. CCS Concepts: • Mathematics of computing → Hypothesis testing and confidence interval computation; • General and reference → Experimentation.

show abstract

Section: Summary and Future Workmentioning

confidence: 99%

Ensure A/B Test Quality at Scale with Automated Randomization Validation and Sample Ratio Mismatch Detection

Nie,

Zhang,

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Online experimentation has been playing a key role in data-driven decision making in the IT industry including Microsoft [16,17], Google [29], Linkedin [33], Netflix [32], Uber, eBay [23], and many others [9]. Generally, online controlled experimentation, also known as A/B testing, is conducted for a pre-determined amount of time to compare the difference in metrics between the treatment group and the control group where users are randomly assigned to.…”

Section: Introductionmentioning

confidence: 99%

Clustering-Based Imputation for Dropout Buyers in Large-Scale Online Experimentation

Shen¹,

Mao²,

Zhang³

et al. 2023

The New England Journal of Statistics in Data Science

View full text Add to dashboard Cite

In online experimentation, appropriate metrics (e.g., purchase) provide strong evidence to support hypotheses and enhance the decision-making process. However, incomplete metrics are frequently occurred in the online experimentation, making the available data to be much fewer than the planned online experiments (e.g., A/B testing). In this work, we introduce the concept of dropout buyers and categorize users with incomplete metric values into two groups: visitors and dropout buyers. For the analysis of incomplete metrics, we propose a clustering-based imputation method using k-nearest neighbors. Our proposed imputation method considers both the experiment-specific features and users’ activities along their shopping paths, allowing different imputation values for different users. To facilitate efficient imputation of large-scale data sets in online experimentation, the proposed method uses a combination of stratification and clustering. The performance of the proposed method is compared to several conventional methods in both simulation studies and a real online experiment at eBay.

show abstract

Ensure A/B Test Quality at Scale with Automated Randomization Validation and Sample Ratio Mismatch Detection

Nie¹,

Zhang²,

Xu³

et al. 2022

Proceedings of the 31st ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

Dealing with Ratio Metrics in A/B Testing at the Presence of Intra-user Correlation and Segments

Cited by 5 publications

References 10 publications

Ensure A/B Test Quality at Scale with Automated Randomization Validation and Sample Ratio Mismatch Detection

Ensure A/B Test Quality at Scale with Automated Randomization Validation and Sample Ratio Mismatch Detection

Clustering-Based Imputation for Dropout Buyers in Large-Scale Online Experimentation

Ensure A/B Test Quality at Scale with Automated Randomization Validation and Sample Ratio Mismatch Detection

Contact Info

Product

Resources

About