Principal Components and the Problem of Multicollinearity

Abstract

Multicollinearity among independent variables within a regress ion model is one of the most frequently encountered problems faced by the applied researcher. In a recent article in this Journal (Willis, e1 a/.) , a catalog of"remedies" for multicollinearity was presented to assist in reducing its level and associated adverse con sequences. One of these remedies-principal components-was suggested as a method oftransforming a set of collinear explanatory variables into new variables that are orthogonal to each other with the first few of these transformed va riables accounting for the majority of the variability in the origina l data set. In principal components regression , a transformed variable is determined to be important a nd included or unimportant and excluded in the regression model depending upon the size of the characteristic root (eigenvalue) associated with its corresponding characteristic vector (eigenvector) (Massy), the statistical significance of its regression coefficient (Mittelhammer and Baritelle), or some combination of eigenvalue size and correlation with the dependent variable (Johnson, et a/.). Unfortunately, this technique is widely abused and misunderstood by the applied researcher. Confusion exists with respect to (I) its relationship to other orthogonalization techniques; (2) the meaning of the orthogonalized components and the necessity of transforming the principal component estimators back to the original parameter space; (3) the implications of deleting components and the correspondence of this technique to a particular type of restricted least squares estimator; (4) the proper way to delete components and evaluate these implied restrictions; and (5) actual implementation of this estimation procedure via available computer routines. The purpose of this note, therefore, is to place the technique of principal components in perspective and to suggest a methodology for implementing this technique correctly. DEALING WITH MULTICOLLINEARITY Multicollinearity is the result of a lack of selective variation among the independent variables in a regression model. It is a problem associated with passively generated data, i.e. , data obtained from some outside source over which the investigator has no control or data characterized by lack of experimental design. Consequently, the problem of multicollinearity can never be cured; it can only be treated in an ad hoc manner.

Identification of Relationships between Cotton Yield, Quality, and Soil Properties

“…The signs and magnitudes in the OLS models may not be reliable, however, because higher VIF values appeared at exchangeable Ca 2+ and texture measurements (Table 4). Simple deletion of these variables might reduce VIF, but would be at the risk of information loss because exchangeable Ca 2+ and soil texture were important soil properties in this region (Morzuch and Ruark, 1991). Furthermore, some soil properties had already been excluded and remaining variables had different meanings in terms of their contributions to crop growth.…”

“…In either case, resultant LCs were orthogonal. The PLS and PCR, however, have been proven to reduce multicollinearity between independent variables greatly in many cases and have advantages over variable deletion (Morzuch and Ruark, 1991; Stenberg, 1998; Tobias, 1999; Ward and Cox, 2000). Furthermore, PLS is also robust against skewness and omission of regressors (Cassel et al, 1999).…”

“…The regression model can indicate the extent by which the dependent variable can be predicted by the independent variables; the strength of prediction is expressed as r 2 , known as variation in dependent variable explained by independent variables or coefficient of determination. The intercorrelation between soil properties does not violate the assumptions of the classical regression modeling, but can cause multicollinearity problems that may result in large variances of parameter estimates, unexplainable signs of explanatory variables, and rejection of regression coefficient estimates that are significant (Morzuch and Ruark, 1991; Neter et al, 1996; Fekedulegn et al, 2002).…”

“…Each method has its own limitations. For example, variable removal can focus on the influential variables, but could lose some information due to reduced number of independent variables (Morzuch and Ruark, 1991). The PLS is designed to construct models that provide effective prediction power (Tobias, 1999).…”

“…In a soil classification study, Seeling et al (1991) reported a better soil class delineation by using PCA. Morzuch and Ruark (1991) demonstrated the merits of PCR over variable deletion for constructing a regression model with the presence of multicollinearity between independent variables in the original data set. Stenberg (1998) indicated that PLS and PCR performed equally well in describing the relationships between soil C, N, and other soil chemical, physical, and biological properties, but PCR produced less interpretable scores and loadings of new principal components than did PLS.…”