Bilevel Optimization with a Lower-level Contraction: Optimal Sample Complexity without Warm-Start

Abstract

We analyze a general class of bilevel problems, in which the upper-level problem consists in the minimization of a smooth objective function and the lower-level problem is to find the fixed point of a smooth contraction map. This type of problems include instances of meta-learning, hyperparameter optimization and data poisoning adversarial attacks. Several recent works have proposed algorithms which warm-start the lower-level problem, i.e. they use the previous lower-level approximate solution as a staring point for the lowerlevel solver. This warm-start procedure allows one to improve the sample complexity in both the stochastic and deterministic settings, achieving in some cases the order-wise optimal sample complexity. We show that without warm-start, it is still possible to achieve order-wise optimal and near-optimal sample complexity for the stochastic and deterministic settings, respectively. In particular, we propose a simple method which uses stochastic fixed point iterations at the lower-level and projected inexact gradient descent at the upper-level, that reaches an ǫ-stationary point using O(ǫ −2 ) and Õ(ǫ −1 ) samples for the stochastic and the deterministic setting, respectively. Compared to methods using warm-start, ours is better suited for meta-learning and yields a simpler analysis that does not need to study the coupled interactions between the upper-level and lower-level iterates.

A Single-Timescale Analysis For Stochastic Approximation With Multiple Coupled Sequences

“…There are also other works that incorporate momentum to accelerate the convergence rate; see e.g., (Chen et al, 2022;Khanduri et al, 2021;Guo and Yang, 2021;. After our initial conference submission, we have also noticed some concurrent works that are relevant to this work; e.g., (Dagreou et al, 2022;Grazzi et al, 2022;Li et al, 2022;Hu et al, 2022). Specifically, (Dagreou et al, 2022) proposed a SBO method with the variance-reduction technique and achieved optimal rate.…”

“…Specifically, (Dagreou et al, 2022) proposed a SBO method with the variance-reduction technique and achieved optimal rate. And (Grazzi et al, 2022) proposed a SBO method that achieves the optimal rate without warm-start. The algorithms in (Dagreou et al, 2022;Grazzi et al, 2022) are not a case of the plain-vanilla SA update discussed in this work and thus its analysis is not applicable to our problem.…”

“…And (Grazzi et al, 2022) proposed a SBO method that achieves the optimal rate without warm-start. The algorithms in (Dagreou et al, 2022;Grazzi et al, 2022) are not a case of the plain-vanilla SA update discussed in this work and thus its analysis is not applicable to our problem. In addition, the multi-block min-max bilevel algorithm has been studied in (Hu et al, 2022).…”