Distributed testing and estimation under sparse high dimensional models

Battey, Heather; Fan, Jianqing; Liu, Han; Lu, Junwei; Zhu, Ziwei

doi:10.1214/17-aos1587

Cited by 188 publications

(131 citation statements)

References 26 publications

Supporting

Mentioning

124

Contrasting

Order By: Relevance

“…In addition, when G T ( ) fails to detect violation of the null hypothesis on the whole data set, G T ( ) k may still identify the violation with high power. This was observed in our application to the SEER data, and also echoes the findings in Battey et al (2018). This also suggests that, even when the data set is not huge, it might be desirable to partition the data and examine the partitions for possibly masked violations of the null hypothesis.…”

supporting

confidence: 86%

See 1 more Smart Citation

An online updating approach for testing the proportional hazards assumption with streams of survival data

et al. 2019

View full text Add to dashboard Cite

The Cox model—which remains the first choice for analyzing time‐to‐event data, even for large data sets—relies on the proportional hazards (PH) assumption. When survival data arrive sequentially in chunks, a fast and minimally storage intensive approach to test the PH assumption is desirable. We propose an online updating approach that updates the standard test statistic as each new block of data becomes available and greatly lightens the computational burden. Under the null hypothesis of PH, the proposed statistic is shown to have the same asymptotic distribution as the standard version computed on an entire data stream with the data blocks pooled into one data set. In simulation studies, the test and its variant based on most recent data blocks maintain their sizes when the PH assumption holds and have substantial power to detect different violations of the PH assumption. We also show in simulation that our approach can be used successfully with “big data” that exceed a single computer's computational resources. The approach is illustrated with the survival analysis of patients with lymphoma cancer from the Surveillance, Epidemiology, and End Results Program. The proposed test promptly identified deviation from the PH assumption, which was not captured by the test based on the entire data.

show abstract

supporting

confidence: 86%

“…This was observed in our application to the SEER data, and also echoes the findings in Battey et al . (). This also suggests that, even when the data set is not huge, it might be desirable to partition the data and examine the partitions for possibly masked violations of the null hypothesis.…”

Section: Discussionmentioning

confidence: 97%

An online updating approach for testing the proportional hazards assumption with streams of survival data

et al. 2019

View full text Add to dashboard Cite

show abstract

“…In this way, we are able to break a large-scale computation problem into many small pieces, then solve them with divide-and-conquer procedures and communicate only certain summary statistics. In recent years, distributed statistical inference has received considerable attention, covering a wide range of topics including M-estimation (Chen and Xie, 2014;Rosenblatt and Nadler, 2016;Lee et al, 2017;Battey et al, 2018;Shi, Lu, and Song, 2018;Jordan et al, 2018;Banerjee, Durot, and Sen, 2019;Fan, Guo, and Wang, 2019), hypothesis test (Lalitha, Sarwate, and Javidi, 2014;Battey et al, 2018), confidence intervals (Jordan, Lee, and Yang, 2018;Chen, Liu, and Zhang, 2018;Dobriban and Sheng, 2018;Wang et al, 2019), principal component analysis (Garber, Shamir, and Srebro, 2017;, nonparametric regression (Zhang, Duchi, and Wainwright, 2015;Chang, Lin, and Zhou, 2017;Shang and Cheng, 2017;Han et al, 2018;Szabó and Van Zanten, 2019), Bayesian methods (Xu et al, 2014;Jordan et al, 2018), quantile regression (Volgushev, Chao, and Cheng, 2019;Chen, Liu, and Zhang, 2019), bootstrap inference (Kleiner et al, 2014;Han and Liu, 2016), and so on.…”

Section: Introductionmentioning

confidence: 99%

Communication-Efficient Distributed Linear Discriminant Analysis for Binary Classification

Li¹,

Zhao²

2022

STAT SINICA

View full text Add to dashboard Cite

“…After the usage is over, the EMA releases the memory and puts the partial information back to the hard disk. For an extremely big data set which cannot be loaded to memory of a single processor, a common solution in computer science is to partition the data set into a number of subsets by parallel or cluster computa- (Battey, et. al, 2018;Lin and Xi, 2011;Meeker and Hong, 2014).…”

Section: Introductionmentioning

confidence: 99%

Accounting for Factor Variables in Big Data Regression

Zhang¹,

Yang²

2020

STAT SINICA

View full text Add to dashboard Cite

Continuous and factor explanatory variables are both important in linear regression. To fit a linear model with factor variables, the traditional implementation of the least squares approach is to define a number of dummy variables.This approach is difficult to apply to big data since the size of the design matrix can be significantly inflated by a factor variable, even if the number of factor levels is only moderately large. By treating the factor variable as an index, the article proposes a new approach, called the index least squares approach, to overcome the difficulty. Combining it with the technique of scanning data by rows, the index least squares approach can provide exact solutions to a group of linear models with factor variables simultaneously. Therefore, it avoids the memory barrier caused by the size of the design matrix. As the memory needed is unrelated to the number of observations, the index least squares approach can be used, even if the size of a massive data set is hundreds of times higher than the memory size of the computing system.It is important to study theoretical connections between our proposed approach and the traditional implementation of the least squares approach.We assume that the traditional implementation is carried out by a supercomputer such that (2.2) can always be applied even if X is very large. We want to show that results given by our approach are identical to those given by the traditional implementation.We use (γ 1 , · · · ,γ I ,δ) andσ 2 to denote those from our approach. We use (γ T 1 , · · · ,γ T I ,δ T ) andσ 2 T to denote those from the traditional implementation, derived under (4.2) with X given by (4.3). We use (α T ,ω T 2 , · · · ,ω T I ,δ T ) to denote those from the traditional implementation under (2.5) with X given by (2.6). Then,α T =γ T 1 andω T i =γ T i −γ T 1 for i = 2, · · · , I.The variance-covariance matrices can be formulated respectively. We use F * and F * T to represent the test statistics in the comparison between (4.18) and (4.19) from our and the traditional approaches, respectively.Theorem 1.γ i =γ T i for i = 1, · · · , I,δ =δ T , andσ 2 =σ 2 T .Proof: By the fact that X ⊤ X, X ⊤ y, and y ⊤ y given by (4.3) and (4.13) are identical, respectively, we can show the conclusion by simply studying Statistica Sinica: Newly accepted Paper (accepted author-version subject to English editing) Relation to Traditional Implementation24the formulations of (4.15) and (4.16) with those given by the traditional implementation.andProof: We draw the conclusion by comparing each of them based on those given by Section 4.1 with those given by the traditional implementation.Cov(α T ,δ T ) = Cov(γ 1 ,δ), and Cov(ω T i ,δ T ) = Cov(γ i ,δ) − Cov(γ 1 ,δ), for i, i ′ = 2, · · · , I with i ̸ = i ′ .

show abstract

Distributed testing and estimation under sparse high dimensional models

Cited by 188 publications

References 26 publications

An online updating approach for testing the proportional hazards assumption with streams of survival data

An online updating approach for testing the proportional hazards assumption with streams of survival data

Communication-Efficient Distributed Linear Discriminant Analysis for Binary Classification

Accounting for Factor Variables in Big Data Regression

Contact Info

Product

Resources

About