I-LAMM for sparse learning: Simultaneous control of algorithmic complexity and statistical error

Abstract

We propose a computational framework named iterative local adaptive majorize-minimization (I-LAMM) to simultaneously control algorithmic complexity and statistical error when fitting high dimensional models. I-LAMM is a two-stage algorithmic implementation of the local linear approximation to a family of folded concave penalized quasi-likelihood. The first stage solves a convex program with a crude precision tolerance to obtain a coarse initial estimator, which is further refined in the second stage by iteratively solving a sequence of convex programs with smaller precision tolerances. Theoretically, we establish a phase transition: the first stage has a sublinear iteration complexity, while the second stage achieves an improved linear rate of convergence. Though this framework is completely algorithmic, it provides solutions with optimal statistical performances and controlled algorithmic complexity for a large family of nonconvex optimization problems. The iteration effects on statistical errors are clearly demonstrated via a contraction property. Our theory relies on a localized version of the sparse/restricted eigenvalue condition, which allows us to analyze a large family of loss and penalty functions and provide optimality guarantees under very weak assumptions (For example, I-LAMM requires much weaker minimal signal strength than other procedures). Thorough numerical results are provided to support the obtained theory.

Adaptive Huber regression on Markov-dependent data

“…It is a simplified version of (Sun, Zhou and Fan, 2019, Definition 2). Another LRE condition in ℓ 2 -neighborhood has found applications in (Fan, Liu, Sun and Zhang, 2018b, Definition 4.1).…”

Hard thresholding regression

“…where = ( 1 , … , n ) T . Then, combining Proposition 3.5 and Proposition 4.5 in the work of Fan et al 2018, with c = 0, 0 = 1∕4, and 0 = 1∕4, implies the desired result. It remains to prove (A1).…”

“…Such condition and its variants have been frequently employed in the literature; see the works of Bickel, Ritov, and Tsybakov (2009) ;Raskutti, Wainwright, and Yu (2010); Negahban, Ravikumar, Wainwright, and Yu (2012); Loh and Wainwright (2015), among others. This condition was generalized to its localized version by Fan et al (2018) for general loss functions.…”

“…Zhang () and Huang and Zhang () analyzed the multistage convex relaxation for sparse regularization in the setting of linear models and generalized linear models, respectively. Fan, Liu, Sun, and Zhang () studied the statistical and computational tradeoffs for the general nonconvex sparse learning problems under the weakest possible conditions.…”

“…and Huang and Zhang (2012) analyzed the multistage convex relaxation for sparse regularization in the setting of linear models and generalized linear models, respectively. Fan, Liu, Sun, and Zhang (2018) studied the statistical and computational tradeoffs for the general nonconvex sparse learning problems under the weakest possible conditions. Instead of introducing a different penalty function, we design a procedure with a new loss function that performs similarly as the 0 -penalized regression when only the Lasso penalty is used.…”