A Geometrical Interpretation of Collinearity: A Natural Way to Justify Ridge Regression and Its Anomalies

Abstract

Summary Justifying ridge regression from a geometrical perspective is one of the main contributions of this paper. To the best of our knowledge, this question has not been treated previously. This paper shows that ridge regression is a particular case of raising procedures that provide greater flexibility by transforming the matrix X associated with the model. Thus, raising procedures, based on a geometrical idea of the vectorial space associated with the columns of matrix X, lead naturally to ridge regression and justify the presence of the well‐known constant k on the main diagonal of matrix X′X. This paper also analyses and compares different alternatives to raising with respect to collinearity mitigation. The results are illustrated with an empirical application.

The VIF and MSE in Raise Regression

“…Thus, it will be concluded that an unique application of the raise regression does not guarantee the mitigation of the multicollinearity. Consequently, this extension complements the results presented by García et al [31] and determines, on the one hand, whether it is necessary to apply a successive raise regression (see García et al [31] for more details) and, on the other hand, the most adequate variable for raising and the most optimal value for the raising factor in order to guarantee the mitigation of the multicollinearity.…”

“…Thus, the selection of an adequate value for λ is essential, analogously to what occurs with the ridge factor K. A preliminary proposal about how to select the raising factor in a model with two independent standardized variables can be found in García et al [41]. Other recently published papers introduce and highlight the various advantages of raise estimators for statistical analysis: Salmerón et al [30] presented the raise regression for p = 3 standardized variables and showed that it presents better properties than the ridge regression and that the individual inference of the raised variable is not altered, García et al [31] showed that it is guaranteed that all the VIFs associated with the model in Equation (5) diminish but that it is not possible to quantify the decrease, García and Ramírez [42] presented the successive raise regression, and García et al [31] showed, among other questions, that ridge regression is a particular case of raise regression. Figure 1.…”

“…This paper presents the extension of the VIF to the raise regression showing that, although García et al [31] showed that the application of the raise regression guarantees a diminishing of the VIF, it is not guaranteed that its value is lower the threshold traditionally established as troubling. Thus, it will be concluded that an unique application of the raise regression does not guarantee the mitigation of the multicollinearity.…”

“…It consists in separating the independent variables by using the residuals (weighted by the raising factor) of the auxiliary regression traditionally used to obtain the VIF. Salmerón et al [30] showed that the raise regression presents better conditions than ridge regression and, more recently, García et al [31] showed, among other questions, that the ridge regression is a particular case of the raise regression.…”

“…. , p. If none of the p − 1 asymptotes is lower than the established threshold, it will not be enough to raise one independent variable and a successive raise regression will be recommended (see García and Ramírez [42] and García et al [31] for more details). Note that, if it were necessary to raise more than one variable, it is guaranteed that there will be values of the raising parameter that mitigate multicollinearity since, in the extreme case where all the variables of the model are raised, all the VIFs associated with the raised variables tend to one.…”