Beyond Backpropagation: Bilevel Optimization Through Implicit Differentiation and Equilibrium Propagation

Zucchet, Nicolas; Sacramento, João

doi:10.1162/neco_a_01547

Cited by 6 publications

(1 citation statement)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This approximation occurs as the strength of the nudging becomes infinitesimal, resulting the difference between the phases resembling a finite-difference gradient [36]. Other efforts have developed theory for EP, casting it in terms of a type of bilevel optimization [85]. However, the original formulations of CHL schemes such as EP generally fail to scale up to complex tasks; to address this issue, variations of this CHL/EP have been designed that improve efficiency as well as performance on particular tasks [24,70,27].…”

Section: Contrastive Hebbian Learningmentioning

confidence: 99%

A Review of Neuroscience-Inspired Machine Learning

Ororbia,

Mali,

Kohan

et al. 2024

Preprint

View full text Add to dashboard Cite

One major criticism of deep learning centers around the biological implausibility of the credit assignment schema used for learning -- backpropagation of errors. This implausibility translates into practical limitations, spanning scientific fields, including incompatibility with hardware and non-differentiable implementations, thus leading to expensive energy requirements. In contrast, biologically plausible credit assignment is compatible with practically any learning condition and is energy-efficient. As a result, it accommodates hardware and scientific modeling, e.g. learning with physical systems and non-differentiable behavior. Furthermore, it can lead to the development of real-time, adaptive neuromorphic processing systems. In addressing this problem, an interdisciplinary branch of artificial intelligence research that lies at the intersection of neuroscience, cognitive science, and machine learning has emerged. In this paper, we survey several vital algorithms that model bio-plausible rules of credit assignment in artificial neural networks, discussing the solutions they provide for different scientific fields as well as their advantages on CPUs, GPUs, and novel implementations of neuromorphic hardware. We conclude by discussing the future challenges that will need to be addressed in order to make such algorithms more useful in practical applications.

show abstract

Section: Contrastive Hebbian Learningmentioning

confidence: 99%

A Review of Neuroscience-Inspired Machine Learning

Ororbia,

Mali,

Kohan

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

Predictive processing in neuroscience, computational modeling and psychology

Brucklacher,

Lee,

Moreni

et al. 2025

Encyclopedia of the Human Brain

View full text Add to dashboard Cite

Analyzing inexact hypergradients for bilevel learning

Ehrhardt,

Roberts

2023

IMA Journal of Applied Mathematics

View full text Add to dashboard Cite

Estimating hyperparameters has been a long-standing problem in machine learning. We consider the case where the task at hand is modeled as the solution to an optimization problem. Here the exact gradient with respect to the hyperparameters cannot be feasibly computed and approximate strategies are required. We introduce a unified framework for computing hypergradients that generalizes existing methods based on the implicit function theorem and automatic differentiation/backpropagation, showing that these two seemingly disparate approaches are actually tightly connected. Our framework is extremely flexible, allowing its subproblems to be solved with any suitable method, to any degree of accuracy. We derive a priori and computable a posteriori error bounds for all our methods and numerically show that our a posteriori bounds are usually more accurate. Our numerical results also show that, surprisingly, for efficient bilevel optimization, the choice of hypergradient algorithm is at least as important as the choice of lower-level solver.

show abstract

Beyond Backpropagation: Bilevel Optimization Through Implicit Differentiation and Equilibrium Propagation

Cited by 6 publications

References 34 publications

A Review of Neuroscience-Inspired Machine Learning

A Review of Neuroscience-Inspired Machine Learning

Predictive processing in neuroscience, computational modeling and psychology

Analyzing inexact hypergradients for bilevel learning

Contact Info

Product

Resources

About