Applying the Delta Method in Metric Analytics

Abstract

During the last decade, the information technology industry has adopted a data-driven culture, relying on online metrics to measure and monitor business performance. Under the setting of big data, the majority of such metrics approximately follow normal distributions, opening up potential opportunities to model them directly without extra model assumptions and solve big data problems via closed-form formulas using distributed algorithms at a fraction of the cost of simulation-based procedures like bootstrap. However, certain attributes of the metrics, such as their corresponding data generating processes and aggregation levels, pose numerous challenges for constructing trustworthy estimation and inference procedures. Motivated by four real-life examples in metric development and analytics for large-scale A/B testing, we provide a practical guide to applying the Delta method, one of the most important tools from the classic statistics literature, to address the aforementioned challenges. We emphasize the central role of the Delta method in metric analytics by highlighting both its classic and novel applications.

COVID-19 Pandemic Worry and Vaccination Intention: The Mediating Role of the Health Belief Model Components

“…We used the jAMM module, which applies the maximum likelihood estimation method, an optimal procedure for parameter estimations. Using the Delta method, which extends the approximations from the central limit theorem ( Deng et al, 2018 ), we calculated the confidence intervals.…”

Zero to Hero: Exploiting Null Effects to Achieve Variance Reduction in Experiments with One-sided Triggering

“…In fact, 𝜏 𝑡𝑟𝑔1 is unbiased for ITT as long as the augmentation τ0 equals zero in expectation, and this is testable. For example, via a Wald test, using the delta method [5,33] to compute the variance of τ0 , or using bootstrapped Var( τ0 ), as described in Section 2.2.1.…”

“…The extra efficiency gain for the One-Sided Trigger likely comes from its exploitation of weak principal ignorability, which allows it to use the entire control group (i.e., a larger sample size) to obtain a more precise estimate of the control outcome mean for the 𝐶0 group. In our simulation setup, only 25% of units are 5 Naive:…”

The equivalence of the Delta method and the cluster-robust variance estimator for the analysis of clustered randomized experiments

“…On the other camp, the A/B testing or online experimentation literature promotes the delta method [Kohavi et al, 2010, Deng et al, 2017, 2018, which tackles the variance estimation of the ATE estimator directly using large sample theory [ Van der Vaart, 2000, Dasgupta, 2008. The delta method is often presented as a way to extend asymptotic normality of a random variable (or vector of random variables) to a continuous function of the said variable(s).…”

“…The econometrics literature promotes the cluster-robust variance estimator [Athey and Imbens, 2017], which can be dated back to the study of linear regression with clustered residuals [Liang and Zeger, 1986]. The A/B testing or online experimentation literature promotes the delta method [Kohavi et al, 2010, Deng et al, 2017, 2018, which tackles the variance estimation of the ATE estimator directly using large sample theory. The two methods are seemly different as the former begins with a regression setting at the individual unit level and the latter is semi-parametric with only i.i.d.…”