When does more regularization imply fewer degrees of freedom? Sufficient conditions and counterexamples

Abstract

Regularization aims to improve prediction performance of a given statistical modeling approach by moving to a second approach which achieves worse training error but is expected to have fewer degrees of freedom, i.e., better agreement between training and prediction error. We show here, however, that this expected behavior does not hold in general. In fact, counter examples are given that show regularization can increase the degrees of freedom in simple situations, including lasso and ridge regression, which are the most common regularization approaches in use. In such situations, the regularization increases both training error and degrees of freedom, and is thus inherently without merit. On the other hand, two important regularization scenarios are described where the expected reduction in degrees of freedom is indeed guaranteed: (a) all symmetric linear smoothers, and (b) linear regression versus convex constrained linear regression (as in the constrained variant of ridge regression and lasso). *

Degrees of freedom and model search

“…Second, even in situations in which the Lagrange and constrained forms of a particular optimization problem are equivalent (e.g., this is true under strong duality, and so it is true for most convex problems, under very weak conditions), there is a difference between studying the degrees of freedom of an estimator defined in one problem form versus the other. This is because the map from the Lagrange parameter in one form to the constraint bound in the other generically depends on y, i.e., it is a random mapping (Kaufman & Rosset (2013) discuss this for ridge regression and the lasso). Lastly, in this paper, we focus on the Lagrange form (4) of subset selection because we find this problem is easier to analyze mathematically.…”

“…It is worth mentioning the interesting, recent works of Kaufman & Rosset (2013) and Janson et al (2013), which investigate unexpected nonmonoticities in the (total) degrees of freedom of an estimator, as a function of some underlying parametrization for the amount of imposed regularization. We note that the right panel of Figure 3 portrays a definitive example of this, in that the best subset selection degrees of freedom undergoes a major nonmonoticity at 10 (expected) active variables, as discussed above.…”

Effective degrees of freedom: a flawed metaphor

“…The common intuition that "effective" or "equivalent" degrees of freedom serves a consistent and interpretable measure of model complexity merits some degree of skepticism. Our results and examples, combined with those of Kaufman and Rosset (2014), demonstrate that for many widely-used convex and non-convex fitting techniques, the DF can be non-monotone with respect to model nesting. In the nonconvex case, the DF can exceed the dimension of the model space by an arbitrarily large amount.…”

“…Surprisingly, monotonicity can even break down for methods projecting onto convex sets, including ridge regression and the Lasso (although the DF cannot exceed the dimension of the convex set). The non-monotonicity of DF for such convex methods was discovered independently by Kaufman and Rosset (2014), who give a thorough account. Among other results, they prove that the degrees of freedom of projection onto any convex set must always be smaller than the dimension of that set.…”

On Degrees of Freedom of Projection Estimators With Applications to Multivariate Nonparametric Regression

“…In the recent papers Kaufman and Rosset (2014) and Janson et al (2015) The paper is organized as follows. In Section 2 we provide some basic results on the divergence of projection estimators.…”