Expanding boundaries of Gap Safe screening

Abstract

Sparse optimization problems are ubiquitous in many fields such as statistics, signal/image processing and machine learning. This has led to the birth of many iterative algorithms to solve them. A powerful strategy to boost the performance of these algorithms is known as safe screening: it allows the early identification of zero coordinates in the solution, which can then be eliminated to reduce the problem's size and accelerate convergence. In this work, we extend the existing Gap Safe screening framework by relaxing the global strong-concavity assumption on the dual cost function. Instead, we exploit local regularity properties, that is, strong concavity on well-chosen subsets of the domain. The non-negativity constraint is also integrated to the existing framework. Besides making safe screening possible to a broader class of functions that includes β-divergences (e.g., the Kullback-Leibler divergence), the proposed approach also improves upon the existing Gap Safe screening rules on previously applicable cases (e.g., logistic regression). The proposed general framework is exemplified by some notable particular cases: logistic function, β = 1.5 and Kullback-Leibler divergences. Finally, we showcase the effectiveness of the proposed screening rules with different solvers (coordinate descent, multiplicativeupdate and proximal gradient algorithms) and different data sets (binary classification, hyperspectral and count data).

High Dimensional Portfolio Selection with Cardinality Constraints

“…Fortunately, the screening techniques developed for the convex optimization problem can be used to alleviate this problem. Building on the idea of Dantas et al (2021), we propose safe screening methods for the problem (5) and further derive efficient primal updates. We only need the following common assumption about the objective function h(w) for the remaining context to establish our theoretical results.…”

Safe Screening for Logistic Regression with $\ell_0$-$\ell_2$ Regularization

“…Strong rules (Tibshirani, 2011), on the other hand, are heuristics with no guarantee but able to prune a large number of features fast. A large body of work exists on screening rules for 1 -regularized regression (Wang et al, 2013;Liu et al, 2014;Fercoq et al, 2015;Ndiaye et al, 2017;Dantas et al, 2021), including some for logistic regression (Wang et al, 2014). However, little attention has been given to the 0 -regularized regression problem, where dimension reduction by screening rules can have substantially larger impact due to the higher computational burden for solving the non-convex regression problems.…”