Distributed Statistical Inference for Massive Data

Abstract

This paper considers distributed statistical inference for a general type of statistics that encompasses the U-statistics and the M-estimators in the context of massive data where the data can be stored at multiple platforms at different locations. In order to facilitate effective computation and to avoid expensive data communication among different platforms, we formulate distributed statistics which can be computed over smaller data blocks. The statistical properties of the distributed statistics are investigated in terms of the mean square error of estimation and their asymptotic distributions with respect to the number of data blocks. In addition, we propose two distributed bootstrap algorithms which are computationally effective and are able to capture the underlying distribution of the distributed statistics. Numerical simulation and real data applications of the proposed approaches are provided to demonstrate the empirical performance.

Divide and conquer for accelerated failure time model with massive time‐to‐event data

“…In recent years, two methods have commonly been used to tackle the challenges arising due to massive data. One is the divide-and-conquer (DAC) algorithm (e.g., Zhang, Duchi & Wainwright, 2013;Chen & Xie, 2014;Battey et al, 2018;Chen & Peng, 2018); the other is the resampling-based method (e.g., Kleiner et al, 2014;Sengupta, Volgushev & Shao, 2016;Wang, Yang & Stufken, 2018). By the DAC method, a massive dataset is partitioned into small subsamples, and estimators obtained from each subsample are then aggregated to form the final estimator.…”

“…For example, Zhang, Duchi & Wainwright (2013) and Huang & Huo (2019) used DAC for M-estimators; Chen & Xie (2014) and Lee et al (2017) applied DAC to the linear and generalized linear models. Battey et al (2018) used DAC to study hypothesis testing and parameter estimation in a general likelihood-based framework in both low-dimensional and sparse high-dimensional settings; Chen & Peng (2018) applied DAC to 𝑈 -statistics and M-estimators. Chen & Peng (2018) pointed out that the resampling-based method has some limitations, such as high computational cost when the massive data are stored at different locations.…”

“…Battey et al (2018) used DAC to study hypothesis testing and parameter estimation in a general likelihood-based framework in both low-dimensional and sparse high-dimensional settings; Chen & Peng (2018) applied DAC to 𝑈 -statistics and M-estimators. Chen & Peng (2018) pointed out that the resampling-based method has some limitations, such as high computational cost when the massive data are stored at different locations. Additionally, the DAC strategy has been extended to a sparse Cox regression by Wang et al (2019); Xue et al (2020) developed a DAC algorithm that updates test statistics for hypothesis testing of the proportional hazards assumption under the Cox model as blocks of data are received sequentially.…”

Hypothesis Testing of One-Sample Mean Vector in Distributed Frameworks

“…The divide and conquer technique is a popular method in distributed frameworks, where one constructs a statistic or an estimator using data in each machine, and then transmits them to the hub to get a pooled one. The divide and conquer technique has been applied successfully in many problems, including regression and classification, hypothesis testing, confidence intervals, principal eigenspaces analysis, linear discriminant analysis, and many others (Zhang et al, 2013;Hsieh et al, 2014;Zhang et al, 2015;Lin et al, 2017;Szabó and Van Zanten, 2019;Battey et al, 2018;Guo et al, 2019;Chen and Peng, 2018;Jordan et al, 2019;Fan et al, 2019;Tian and Gu, 2017;Li and Zhao, 2020;Dobriban and Sheng, 2021, etc. ).…”