Simultaneous Variable Selection

Abstract

We propose a new method for selecting a common subset of explanatory variables where the aim is to model several response variables. The idea is a natural extension of the LASSO technique proposed by Tibshirani (1996) and is based on the ( joint) residual sum of squares while constraining the parameter estimates to lie within a suitable polyhedral region. The properties of the resulting convex programming problem are analyzed for the special case of an orthonormal design. For the general case, we develop an efficient interior point algorithm. The method is illustrated on a dataset with infrared spectrometry measurements on 14 qualitatively different but correlated responses using 770 wavelengths. The aim is to select a subset of the wavelengths suitable for use as predictors for as many of the responses as possible.

Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems

“…In some signal processing applications, however, the number of nonzero x components may be significant, and since these methods require at least as many pivot operations as there are nonzeros in the solution, they may be less competitive on such problems. The interior-point (IP) approach in [58], which solves a generalization of (4), also requires explicit construction of A T A, though the approach could in principle modified to allow iterative solution of the linear system at each primal-dual iteration.…”

Variable selection in general multinomial logit models

“…For example, the methodology that was developed for multivariate linear regression in Turlach et al (2005) could be extended to multinomial logit models, even though the technical details would be cumbersome. Overall, each penalty term that yields direct variable selection in multinomial logit models must penalize the parameter vectors β •j in a groupwise fashion.…”

Learning a common substructure of multiple graphical Gaussian models

“…as a regularization term analogous to the Group-Lasso (Yuan and Lin, 2006;Bach, 2008) and Multitask-Lasso (Turlach et al, 2005;Liu et al, 2009) with p ∈ [1, ∞]. Varoquaux et al (2010) has considered the case p = 2 while Honorio and Samaras (2010) used p = ∞.…”

“…Another way is to impose general and mild assumptions on the data. This kind of approach is especially common in the multi-task learning literatures (Caruana, 1997;Turlach et al, 2005), where the relationships among datasets are treated as a clue for combining multiple tasks into a single problem. The scope of the present paper is in the latter context where the relationship among datasets is the objective we want to analyze.…”

“…There are some prior studies on learning a set of GGMs from multiple datasets. Varoquaux et al (2010) and Honorio and Samaras (2010) imported the idea of Group-Lasso (Yuan and Lin, 2006;Bach, 2008) and Multitask-Lasso (Turlach et al, 2005;Liu et al, 2009), and extended the framework of a single GGM setting. In both cases, the problem is formulated under the assumption that all matrices share the same zero patterns.…”