Information-Based Optimal Subdata Selection for Big Data Linear Regression

Abstract

Extraordinary amounts of data are being produced in many branches of science. Proven statistical methods are no longer applicable with extraordinary large data sets due to computational limitations. A critical step in big data analysis is data reduction. Existing investigations in the context of linear regression focus on subsampling-based methods. However, not only is this approach prone to sampling errors, it also leads to a covariance matrix of the estimators that is typically bounded from below by a term that is of the order of the inverse of the subdata size. We propose a novel approach, termed information-based optimal subdata selection (IBOSS). Compared to leading existing subdata methods, the IBOSS approach has the following advantages: (i) it is significantly faster; (ii) it is suitable for distributed parallel computing; (iii) the variances of the slope parameter estimators converge to 0 as the full data size increases even if the subdata size is fixed, i.e., the convergence rate depends on the full data size; (iv) data analysis for IBOSS subdata is straightforward and the sampling distribution of an IBOSS estimator is easy to assess. Theoretical results and extensive simulations demonstrate that the IBOSS approach is superior to subsampling-based methods, sometimes by orders of magnitude. The advantages of the new approach are also illustrated through analysis of real data.

Divide-and-Conquer Information-Based Optimal Subdata Selection Algorithm

“…Theorem 1 is aligned with Theorem 3 of Wang et al (2019) for the original IBOSS algorithm, which shows that for the subdata D S obtained from Algorithm 1, the following inequality holds.…”

“…To extract some useful information from the data in time with limited computing resources, a subset of the full data can be selected and thoroughly analyzed. For this purpose, Wang et al (2019) proposed the IBOSS method. We summarize the motivation and procedure in the following.…”

“…To extract the maximum amount of information from the full data with a fixed subset sample size k so that |δ| = N i=1 δ i = k, Wang et al (2019) proposed to select the subset of data that maximizes M δ with respect to δ under the the D-optimality criterion in optimal design of experiments, namely, to find…”

“…The data points obtained according to the p covariates are then combined to form a subdata of size k for statistical analysis. The original IBOSS paper by Wang et al (2019) did not present all practical details of implementing the IBOSS procedure. For example, it is mentioned that if some data points have been included in the subdata by some covariate, the IBOSS algorithm requires to exclude them from consideration when using other covariates to select data points, but the implementation details were not provided.…”

“…The aforementioned work uses random subsampling to take subsets of full data, and Wang et al (2019) showed that an estimator obtained from this approach for linear regression has a variance that is bounded from below by a term that is proportional to the inverse of the subset sample size, i.e., the estimator will not converge to the true parameter if the subset sample size is fixed no matter how fast the full data sample size goes to infinity.…”