Sparse High-Dimensional Regression: Exact Scalable Algorithms and Phase Transitions

Bertsimas, Dimitris; Parys, Bart Van

doi:10.48550/arxiv.1709.10029

Cited by 13 publications

(40 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that inequality (8) is in general weak for bounded continuous variables (as non-negativity or other bounds can be used to strengthen the inequalities, see [3] for additional discussion), and inequality ( 7) is in general weak for arbitrary matrices A T ∈ S T + . Nonetheless, as we now show, inequality (7) gives the ideal description for Q T if A T is a rank-one matrix. Define…”

Section: Convexificationmentioning

confidence: 69%

“…Let (z, β, t) ∈ Q T , and we verify that inequality (7) is satisfied. First observe that if z = 0, then β = 0 and inequality (7) reduces to 0 ≤ t, which is satisfied. Otherwise, if z i = 1 for some i ∈ T , then z(T ) ≥ 1 and we find that…”

Section: Convexificationmentioning

confidence: 99%

“…Problem (1) encompasses a broad range of the regression models. It includes as special cases: ridge regression [22], when λ > 0, µ = 0 and k ≥ p; lasso [34], when λ = 0, µ ≥ 0 and k ≥ p; elastic net [44] when λ, µ > 0 and k ≥ p; best subset selection [29], when λ = µ = 0 and k < p. Additionally, Bertsimas and Van Parys [7] propose to solve (1) with λ > 0, µ = 0 and k < p for high-dimensional regression problems, while Mazumder et al [28] study (1) with λ = 0, µ > 0 and k < p for problems with low Signal-to-Noise Ratios (SNR). The results in this paper cover all versions of (1) with k < p; moreover, they can be extended to problems with non-separable regularizations of the form λ Aβ 2 2 + µ Cβ 1 , resulting in sparse variants of the fused lasso [35], generalized lasso [36] and smooth lasso [21], among others.…”

Section: Introductionmentioning

confidence: 99%

“…Pilanci et al [32] exploit the Tikhonov regularization term and convex analysis to construct an improved convex relaxation using the reverse Huber penalty. In a similar vein, Bertsimas and Van Parys [7] leverage the Tikhonov regularization and duality to propose an efficient algorithm for high-dimensional sparse regression.…”

Section: Introductionmentioning

confidence: 99%

“…Moreover, they show that the continuous relaxation of ( 5) is equivalent to the continuous relaxation of the formulation used by Bertsimas and Van Parys [7]. Dong et al [11] also study the perspective relaxation in the context of regression: first, they show that using the reverse Huber penalty [32] is, in fact, equivalent to just solving the convex relaxation of (5) -thus the relaxations of [7,32,38] all coincide; second, they propose to use an optimal perspective relaxation, i.e., by applying the perspective relaxation to a separable quadratic function β Dβ, where D is a nonnegative diagonal matrix such that X X + λI − D 0; finally, they show that solving this stronger convex relaxation of the optimal perspective relaxation is, in fact, equivalent to using the MC + penalty [39]. Among these approaches the optimal perspective relaxation of Dong et al [11] is the only one that does not explicitly require the use of the Tikhonov regularization λ β 2 2 .…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Rank-one Convexification for Sparse Regression

Atamtürk¹,

Gómez²

2019

Preprint

View full text Add to dashboard Cite

Sparse regression models are increasingly prevalent due to their ease of interpretability and superior out-of-sample performance. However, the exact model of sparse regression with an 0 constraint restricting the support of the estimators is a challenging non-convex optimization problem. In this paper, we derive new strong convex relaxations for sparse regression. These relaxations are based on the ideal (convex-hull) formulations for rank-one quadratic terms with indicator variables. The new relaxations can be formulated as semidefinite optimization problems in an extended space and are stronger and more general than the state-of-the-art formulations, including the perspective reformulation and formulations with the reverse Huber penalty and the minimax concave penalty functions. Furthermore, the proposed rankone strengthening can be interpreted as a non-separable, non-convex sparsityinducing regularizer, which dynamically adjusts its penalty according to the shape of the error function. In our computational experiments with benchmark datasets, the proposed conic formulations are solved within seconds and result in near-optimal solutions (with 0.4% optimality gap) for non-convex 0 -problems. Moreover, the resulting estimators also outperform alternative convex approaches from a statistical viewpoint, achieving high prediction accuracy and good interpretability.

show abstract

Section: Convexificationmentioning

confidence: 69%

Section: Convexificationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Rank-one Convexification for Sparse Regression

Atamtürk¹,

Gómez²

2019

Preprint

View full text Add to dashboard Cite

show abstract

Certifiably optimal sparse inverse covariance estimation

Bertsimas¹,

Lamperski²,

Pauphilet³

2019

Math. Program.

Self Cite

View full text Add to dashboard Cite

We consider the maximum likelihood estimation of sparse inverse covariance matrices. We demonstrate that current heuristic approaches primarily encourage robustness, instead of the desired sparsity.We give a novel approach that solves the cardinality constrained likelihood problem to certifiable optimality. The approach uses techniques from mixed-integer optimization and convex optimization, and provides a high-quality solution with a guarantee on its suboptimality, even if the algorithm is terminated early. Using a variety of synthetic and real datasets, we demonstrate that our approach can solve problems where the dimension of the inverse covariance matrix is up to 1, 000s. We also demonstrate that our approach produces significantly sparser solutions than Glasso and other popular learning procedures, makes less false discoveries, while still maintaining state-of-the-art accuracy.

show abstract

MIP-BOOST: Efficient and Effective L₀ Feature Selection for Linear Regression

Kenney

Chiaromonte

Felici

2021

Journal of Computational and Graphical Statistics

View full text Add to dashboard Cite

Because of continuous advances in mathematical programing, Mix Integer Optimization has become a competitive vis-a-vis popular regularization method for selecting features in regression problems. The approach exhibits unquestionable foundational appeal and versatility, but also poses important challenges. We tackle these challenges, reducing computational burden when tuning the sparsity bound (a parameter which is critical for effectiveness) and improving performance in the presence of feature collinearity and of signals that vary in nature and strength. Importantly, we render the approach efficient and effective in applications of realistic size and complexity without resorting to relaxations or heuristics in the optimization, or abandoning rigorous cross-validation tuning. Computational viability and improved performance in subtler scenarios is achieved with a multi-pronged blueprint, leveraging characteristics of the Mixed Integer Programming framework and by means of whitening, a data pre-processing step.

show abstract

Sparse High-Dimensional Regression: Exact Scalable Algorithms and Phase Transitions

Cited by 13 publications

References 0 publications

Rank-one Convexification for Sparse Regression

Rank-one Convexification for Sparse Regression

Certifiably optimal sparse inverse covariance estimation

MIP-BOOST: Efficient and Effective L₀ Feature Selection for Linear Regression

Contact Info

Product

Resources

About

Sparse High-Dimensional Regression: Exact Scalable Algorithms and Phase Transitions

Cited by 13 publications

References 0 publications

Rank-one Convexification for Sparse Regression

Rank-one Convexification for Sparse Regression

Certifiably optimal sparse inverse covariance estimation

MIP-BOOST: Efficient and Effective L0 Feature Selection for Linear Regression

Contact Info

Product

Resources

About

MIP-BOOST: Efficient and Effective L₀ Feature Selection for Linear Regression