Safely and Quickly Deploying New Features with a Staged Rollout Framework Using Sequential Test and Adaptive Experimental Design

Abstract

During the rapid development cycle for Internet products (websites and mobile apps), new features are developed and rolled out to users constantly. Features with code defects or design flaws can cause outages and significant degradation of user experience. The traditional method of code review and change management can be time-consuming and error-prone. In order to make the feature rollout process safe and fast, this paper proposes a methodology for rolling out features in an automated way using an adaptive experimental design. Under this framework, a feature is gradually ramped up from a small proportion of users to a larger population based on real-time evaluation of the performance of important metrics. If there are any regression detected during the rampup step, the ramp-up process stops and the feature developer is alerted. There are two main algorithm components powering this framework: 1) a continuous monitoring algorithm -using a variant of the sequential probability ratio test (SPRT) to monitor the feature performance metrics and alert feature developers when a metric degradation is detected, 2) an automated ramp-up algorithm -deciding when and how to ramp up to the next stage with larger sample size. This paper presents one monitoring algorithm and three ramping up algorithms including time-based, power-based, and riskbased (a Bayesian approach) schedules. These algorithms are evaluated and compared on both simulated data and real data. There are three benefits provided by this framework for feature rollout: 1) for defective features, it can detect the regression early and reduce negative effect, 2) for healthy features, it rolls out the feature quickly, 3) it reduces the need for manual intervention via the automation of the feature rollout process.

Rapid Regression Detection in Software Deployments through Sequential Testing

“…when a regression is detected. Towards this end, sequential tests for difference-in-means of Gaussian random variables have already been widely used for online A/B experiments [11][12][13]27]. However, we argue that performing inference about means is too limited for canary tests, for not all bugs or performance regressions can be captured by differences in the mean alone, as the following example demonstrates.…”

“…Johari et al [11,13] proposed an "always-valid" sequential inference framework for differences in the means of Gaussian random variables using the mSPRT to provide confidence sequences and sequential 𝑝-values. In addition to being used in commercial A/B testing software, [27] use this framework for managing the automated rollout of new software features, formulating performance regressions as differences in the mean.…”

“…Many of the earlier works formulate performance regressions as differences in the mean between arms A and B [27], with a performance regression occurring if the mean shifts in the undesirable direction of that metric. We argue from a decision-theoretic perspective that comparing means alone is insufficient to define a performance regression.…”

Ensure A/B Test Quality at Scale with Automated Randomization Validation and Sample Ratio Mismatch Detection

“…Besides, testing immature features could cause outages or critical harmness on key business metrics. The monitoring on key metrics [20,22,24] can help alarm experimenter at the very early hours and ensure a safe data collection, which can be another interesting application to further explore.…”

“…Both are targeted to minimize false positives in their processes while attaining good recall of the signals. The methodologies involve a novel Population Stability Index (PSI [21]) based test and a sequential probability ratio test (SPRT [4,5,14,18,19,24]). To our knowledge, we are the first to automate the methods (in early 2019) in a large scale experimentation platform to continuously monitor experiment quality.…”