Gradient-based Bi-level Optimization for Deep Learning: A Survey

Chen, Can; Chen, Xi; Ma, Chao; Liu, Zixuan; Li, Xue

doi:10.48550/arxiv.2207.11719

Cited by 6 publications

(5 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Given a good choice of γ, the produced X h is expected to have a high score f aux (X h ), based on which we can choose γ. To make the search for γ more efficient, we can formulate this process as a bi-level optimization problem (Chen et al, 2022a;2022c):…”

Section: Deep Linearization For Bidirectional Learningmentioning

confidence: 99%

Bidirectional Learning for Offline Model-based Biological Sequence Design

Chen¹,

Zhang²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

Offline model-based optimization aims to maximize a black-box objective function with a static dataset of designs and their scores. In this paper, we focus on biological sequence design to maximize some sequence score. A recent approach employs bidirectional learning, combining a forward mapping for exploitation and a backward mapping for constraint, and it relies on the neural tangent kernel (NTK) of an infinitely wide network to build a proxy model. Though effective, the NTK cannot learn features because of its parametrization, and its use prevents the incorporation of powerful pre-trained Language Models (LMs) that can capture the rich biophysical information in millions of biological sequences. We adopt an alternative proxy model, adding a linear head to a pre-trained LM, and propose a linearization scheme. This yields a closed-form loss and also takes into account the biophysical information in the pre-trained LM. In addition, the forward mapping and the backward mapping play different roles and thus deserve different weights during sequence optimization. To achieve this, we train an auxiliary model and leverage its weak supervision signal via a bi-level optimization framework to effectively learn how to balance the two mappings. Further, by extending the framework, we develop the first learning rate adaptation module Adaptive-η, which is compatible with all gradient-based algorithms for offline model-based optimization. Experimental results on DNA/protein sequence design tasks verify the effectiveness of our algorithm. Our code is available here.

show abstract

Section: Deep Linearization For Bidirectional Learningmentioning

confidence: 99%

Bidirectional Learning for Offline Model-based Biological Sequence Design

Chen¹,

Zhang²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In the meta-training phase, meta-learning occurs via casting the problem as a bilevel optimization [6], where the inner optimizaiton uses a set of inner learning algorithms to solve a set of related tasks, minimizing some inner objective. During metalearning, an outer algorithm updates the inner learning algorithm's inductive biases such that the models learn to improve on some pre-determined outer objective.…”

Section: Background and Related Workmentioning

confidence: 99%

Fast and Efficient Local-Search for Genetic Programming Based Loss Function Learning

Raymond

Chen

Xue

et al. 2023

Proceedings of the Genetic and Evolutionary Computation Conference

View full text Add to dashboard Cite

In this paper, we develop upon the topic of loss function learning, an emergent meta-learning paradigm that aims to learn loss functions that significantly improve the performance of the models trained under them. Specifically, we propose a new meta-learning framework for task and model-agnostic loss function learning via a hybrid search approach. The framework first uses genetic programming to find a set of symbolic loss functions. Second, the set of learned loss functions is subsequently parameterized and optimized via unrolled differentiation. The versatility and performance of the proposed framework are empirically validated on a diverse set of supervised learning tasks. Results show that the learned loss functions bring improved convergence, sample efficiency, and inference performance on tabulated, computer vision, and natural language processing problems, using a variety of task-specific neural network architectures. CCS Concepts• Computing methodologies → Machine learning algorithms.

show abstract

“…( 5) may lead to a trivial solution where all hierarchical weights reduce to zeros. In this work, we propose to formulate hierarchical weight learning as bi-level optimization Luca et al (2018); Chen et al (2022a); where the hierarchical weight is decided by an outer level learning task:…”

Section: Hierarchical Weight Learningmentioning

confidence: 99%