On $\ell_p$-hyperparameter Learning via Bilevel Nonsmooth Optimization

Okuno, Takayuki; Kawana, Akihiro; Watanabe, Motokazu

doi:10.48550/arxiv.1806.01520

Cited by 13 publications

(36 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The bilevel optimization has been widely applied to various machine learning applications. Hyper-parameter optimization (Lorraine and Duvenaud 2018;Okuno, Takeda, and Kawana 2018;Franceschi et al 2018) uses bilevel optimization extensively. Besides, the idea of bilevel optimization has also been applied to meta learning (Zintgraf et al 2019;Song et al 2019;Soh, Cho, and Cho 2020), neural architecture search (Liu, Simonyan, and Yang 2018;Wong et al 2018;Xu et al 2019), adversarial learning (Tian et al 2020;Yin et al 2020;Gao et al 2020), deep reinforcement learning (Yang et al 2018;Tschiatschek et al 2019), etc.…”

Section: Related Workmentioning

confidence: 99%

A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inverse

Li¹,

Gu²,

Huang³

2021

Preprint

View full text Add to dashboard Cite

In this paper, we propose a new Hessian inverse free Fully Single Loop Algorithm (FSLA) for bilevel optimization problems. Classic algorithms for bilevel optimization admit a double loop structure which is computationally expensive. Recently, several single loop algorithms have been proposed with optimizing the inner and outer variable alternatively. However, these algorithms not yet achieve fully single loop. As they overlook the loop needed to evaluate the hyper-gradient for a given inner and outer state. In order to develop a fully single loop algorithm, we first study the structure of the hypergradient and identify a general approximation formulation of hyper-gradient computation that encompasses several previous common approaches, e.g. back-propagation through time, conjugate gradient, etc. Based on this formulation, we introduce a new state variable to maintain the historical hyper-gradient information. Combining our new formulation with the alternative update of the inner and outer variables, we propose an efficient fully single loop algorithm. We theoretically show that the error generated by the new state can be bounded and our algorithm converges with the rate of O( −2 ). Finally, we verify the efficacy our algorithm empirically through multiple bilevel optimization based machine learning tasks.

show abstract

Section: Related Workmentioning

confidence: 99%

A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inverse

Li¹,

Gu²,

Huang³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…where ζ t i is the i-th element of ζ t in the t-th fold. The linear programs (LPs) (15), for t = 1, . .…”

Section: Combining With (−Amentioning

confidence: 99%

“…By eliminating λ t and w t with w t = (B t ) ⊤ α t in (4c), we get the reduced KKT conditions for problem (15) with…”

Section: Combining With (−Amentioning

confidence: 99%

“…In terms of selecting hyperparameters through bilevel optimization, different models and approaches have been considered in the literature. For example, Okuno et al [15] propose a bilevel optimization model to select the best hyperparameter for a nonsmooth, possibly nonconvex, l p -regularized problem. They then present a smoothing-type algorithm with convergence analysis to solve this bilevel optimization model.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Bilevel hyperparameter optimization for support vector classification: theoretical analysis and a solution method

Li¹,

Li²,

Zemkoho³

2021

Preprint

View full text Add to dashboard Cite

Support vector classification (SVC) is a classical and well-performed learning method for classification problems. A regularization parameter, which significantly affects the classification performance, has to be chosen and this is usually done by the cross-validation procedure. In this paper, we reformulate the hyperparameter selection problem for support vector classification as a bilevel optimization problem in which the upper-level problem minimizes the average number of misclassified data points over all the cross-validation folds, and the lower-level problems are the l1-loss SVC problems, with each one for each fold in T-fold cross-validation. The resulting bilevel optimization model is then converted to a mathematical program with equilibrium constraints (MPEC). To solve this MPEC, we propose a global relaxation crossvalidation algorithm (GR-CV) based on the well-know Sholtes-type global relaxation method (GRM). It is proven to converge to a Cstationary point. Moreover, we prove that the MPEC-tailored version

show abstract

“…Nevertheless, they showed that the 1 2 -norm has better denoising performance than the 1 -norm. Recently, [23] considered the bilevel program (3) with the function R 1 (ω) := n i=1 |ω i | p (0 < p ≤ 1) (i.e. the p -regularizer) by employing smoothing method via the twice continuously differentiable function…”

Section: Introductionmentioning

confidence: 99%

Unified Smoothing Approach for Best Hyperparameter Selection Problem Using a Bilevel Optimization Strategy

Nguyen¹,

Alcantara²,

Okuno³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Strongly motivated from use in various fields including machine learning, the methodology of sparse optimization has been developed intensively so far. Especially, the recent advance of algorithms for solving problems with nonsmooth regularizers is remarkable. However, those algorithms suppose that weight parameters of regularizers, called hyperparameters hereafter, are pre-fixed, and it is a crucial matter how the best hyperparameter should be selected. In this paper, we focus on the hyperparameter selection of regularizers related to the p function with 0 < p ≤ 1 and

show abstract

On $\ell_p$-hyperparameter Learning via Bilevel Nonsmooth Optimization

Cited by 13 publications

References 31 publications

A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inverse

A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inverse

Bilevel hyperparameter optimization for support vector classification: theoretical analysis and a solution method

Unified Smoothing Approach for Best Hyperparameter Selection Problem Using a Bilevel Optimization Strategy

Contact Info

Product

Resources

About