Accounting for Factor Variables in Big Data Regression

Abstract

Continuous and factor explanatory variables are both important in linear regression. To fit a linear model with factor variables, the traditional implementation of the least squares approach is to define a number of dummy variables.This approach is difficult to apply to big data since the size of the design matrix can be significantly inflated by a factor variable, even if the number of factor levels is only moderately large. By treating the factor variable as an index, the article proposes a new approach, called the index least squares approach, to overcome the difficulty. Combining it with the technique of scanning data by rows, the index least squares approach can provide exact solutions to a group of linear models with factor variables simultaneously. Therefore, it avoids the memory barrier caused by the size of the design matrix. As the memory needed is unrelated to the number of observations, the index least squares approach can be used, even if the size of a massive data set is hundreds of times higher than the memory size of the computing system.It is important to study theoretical connections between our proposed approach and the traditional implementation of the least squares approach.We assume that the traditional implementation is carried out by a supercomputer such that (2.2) can always be applied even if X is very large. We want to show that results given by our approach are identical to those given by the traditional implementation.We use (γ 1 , · · · ,γ I ,δ) andσ 2 to denote those from our approach. We use (γ T 1 , · · · ,γ T I ,δ T ) andσ 2 T to denote those from the traditional implementation, derived under (4.2) with X given by (4.3). We use (α T ,ω T 2 , · · · ,ω T I ,δ T ) to denote those from the traditional implementation under (2.5) with X given by (2.6). Then,α T =γ T 1 andω T i =γ T i −γ T 1 for i = 2, · · · , I.The variance-covariance matrices can be formulated respectively. We use F * and F * T to represent the test statistics in the comparison between (4.18) and (4.19) from our and the traditional approaches, respectively.Theorem 1.γ i =γ T i for i = 1, · · · , I,δ =δ T , andσ 2 =σ 2 T .Proof: By the fact that X ⊤ X, X ⊤ y, and y ⊤ y given by (4.3) and (4.13) are identical, respectively, we can show the conclusion by simply studying Statistica Sinica: Newly accepted Paper (accepted author-version subject to English editing) Relation to Traditional Implementation24the formulations of (4.15) and (4.16) with those given by the traditional implementation.andProof: We draw the conclusion by comparing each of them based on those given by Section 4.1 with those given by the traditional implementation.Cov(α T ,δ T ) = Cov(γ 1 ,δ), and Cov(ω T i ,δ T ) = Cov(γ i ,δ) − Cov(γ 1 ,δ), for i, i ′ = 2, · · · , I with i ̸ = i ′ .