Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis

Abstract

Bilevel optimization has become a powerful framework in a variety of machine learning applications including signal processing, meta-learning, hyperparameter optimization, reinforcement learning and network architecture search. There are generally two classes of bilevel optimization formulations for modern machine learning: 1) problem-based bilevel optimization, whose inner-level problem is formulated as finding a minimizer of a given loss function; and 2) algorithm-based bilevel optimization, whose inner-level solution is an output of a fixed algorithm. For the first problem class, two popular types of gradient-based algorithms have been proposed to estimate the gradient of the outer-level objective (hypergradient) via approximate implicit differentiation (AID) and iterative differentiation (ITD). Algorithms for the second problem class include the popular model-agnostic meta-learning (MAML) and almost no inner loop (ANIL). Although bilevel optimization algorithms have been widely used, their convergence rate and fundamental limitations have not been well explored.

A Fast and Convergent Proximal Algorithm for Regularized Nonconvex and Nonsmooth Bi-level Optimization

“…Applications of Bi-level Optimization. Bi-level optimization has been widely applied to meta-learning (Snell et al, 2017;Franceschi et al, 2018;Rajeswaran et al, 2019;Zügner & Günnemann, 2019;Ji et al, 2020b;Ji, 2021), hyperparameter optimization (Franceschi et al, 2017;Shaban et al, 2019), reinforcement learning (Konda & Tsitsiklis, 2000;Hong et al, 2020), and data poisoning (Mehra et al, 2020). For example, (Snell et al, 2017) reformulated the meta-learning objective function under a shared embedding model into a bi-level optimization problem.…”

Network Utility Maximization with Unknown Utility Functions: A Distributed, Data-Driven Bilevel Optimization Approach

“…In this paper, we consider unknown utility functions and provide a distributed solution from a new bilevel optimization perspective, where the lower-level problem is a standard distributed resource allocation algorithm with parameterized surrogate utility functions such as α−fair utility functions, and the upper-level is to fine-tune the surrogate utility functions based on user experiences/feedback. While the solution is based on bilevel optimization, it is very different from existing studies for non-distributed bilevel optimization [30,31,32,33,34,35,36,37] (see [38] and [39] for a more comprehensive overview) due to the distributed nature of the solution over the communication networks. In addition, these approaches cannot be directly applied here due to the computation of either the Hessian inverse or a product of Hessians of the NUM objective, which requires each node to know the infeasible global network information, which is not practical.…”