Deep Equilibrium Models

Bai, Shaojie; Kolter, J. Zico; Koltun, Vladlen

doi:10.48550/arxiv.1909.01377

Cited by 76 publications

(82 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Motivated by the surprisingly recent works [1,8,9] that employ the same transformation in each layer and still achieve competitive results with the state-of-the-art, Bai et al [2] proposed a new approach to model this process and directly computed the fixed point. To leverage ideas from DEQ, Gilton et al [10] proposed DEQ for inverse problems in imaging, which corresponds to a potentially infinite number of iteration steps in the PnP reconstruction scheme.…”

Section: Deep Equilibrium Modelsmentioning

confidence: 99%

Deep Equilibrium Models for Video Snapshot Compressive Imaging

Zhao¹,

Yuan²

2022

Preprint

View full text Add to dashboard Cite

The ability of snapshot compressive imaging (SCI) systems to efficiently capture high-dimensional (HD) data has led to an inverse problem, which consists of recovering the HD signal from the compressed and noisy measurement. While reconstruction algorithms grow fast to solve it with the recent advances of deep learning, the fundamental issue of accurate and stable recovery remains. To this end, we propose deep equilibrium models (DEQ) for video SCI, fusing data-driven regularization and stable convergence in a theoretically sound manner. Each equilibrium model implicitly learns a nonexpansive operator and analytically computes the fixed point, thus enabling unlimited iterative steps and infinite network depth with only a constant memory requirement in training and testing. Specifically, we demonstrate how DEQ can be applied to two existing models for video SCI reconstruction: recurrent neural networks (RNN) and Plug-and-Play (PnP) algorithms. On a variety of datasets and real data, both quantitative and qualitative evaluations of our results demonstrate the effectiveness and stability of our proposed method. The code and models will be released to the public.

show abstract

Section: Deep Equilibrium Modelsmentioning

confidence: 99%

Deep Equilibrium Models for Video Snapshot Compressive Imaging

Zhao¹,

Yuan²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In cases where the property of the landscape that we are trying to optimize is known and can be phrased in terms of the fixed point of some dynamical system, there can often be significant benefit to computing derivatives implicitly rather than unrolling optimization explicitly Rajeswaran et al [2019], Bai et al [2019], Blondel et al [2021]. Using implicit differentiation in this way removes conditioning issues due to unrolling and improves memory cost of computing derivatives.…”

Section: Well-behaved Proxy Objectivesmentioning

confidence: 99%

Gradients are Not All You Need

Metz¹,

Freeman²,

Schoenholz³

et al. 2021

Preprint

View full text Add to dashboard Cite

Differentiable programming techniques are widely used in the community and are responsible for the machine learning renaissance of the past several decades. While these methods are powerful, they have limits. In this short report, we discuss a common chaos based failure mode which appears in a variety of differentiable circumstances, ranging from recurrent neural networks and numerical physics simulation to training learned optimizers. We trace this failure to the spectrum of the Jacobian of the system under study, and provide criteria for when a practitioner might expect this failure to spoil their differentiation based optimization algorithms. * Equal contribution 2 Up to numerical precision, though see [Chow and Palmer, 1992, Kachman et al., 2017] for cases where the gradients will be close.Preprint. Under review.

show abstract

“…Therefore, as an IGNN forward pass we can iterate (2) K times, where K is sufficiently large, to compute node embeddings Y (K) ≈ Y * for use within the meta-loss from (1). For the IGNN backward pass, implicit differentiation methods (Bai et al, 2019;, carefully tailored for handling graph-based models , can but used to compute gradients of Y * with respect to W p and W x . Critically, this implicit technique does not require storing each intermediate representation {Y (k) } K k=1 from the forward pass, and hence is quite memory efficient even if K is arbitrarily large.…”

Section: Overview Of Implicit Gnnsmentioning

confidence: 99%

“…To address these issues, at least in part, two distinct strategies have recently been proposed. First, the framework of implicit deep learning (Bai et al, 2019; has be applied to producing supervised node embeddings that satisfy an equilibrium criteria instantiated through a graph-dependent fixed-point equation. The resulting so-called implicit GNN (IGNN) pipeline , and related antecedents (Dai et al, 2018;Gallicchio & Micheli, 2020), mimics the behavior of a GNN model with arbitrary depth for handling long-range dependencies, but is nonetheless trainable via a fixed memory budget and robust against oversmoothing effects.…”

Section: Introductionmentioning

confidence: 99%

Implicit vs Unfolded Graph Neural Networks

Yang¹,

Wang²,

Huang³

et al. 2021

Preprint

View full text Add to dashboard Cite

It has been observed that graph neural networks (GNN) sometimes struggle to maintain a healthy balance between the efficient modeling long-range dependencies across nodes while avoiding unintended consequences such oversmoothed node representations or sensitivity to spurious edges. To address this issue (among other things), two separate strategies have recently been proposed, namely implicit and unfolded GNNs. The former treats node representations as the fixed points of a deep equilibrium model that can efficiently facilitate arbitrary implicit propagation across the graph with a fixed memory footprint. In contrast, the latter involves treating graph propagation as unfolded descent iterations as applied to some graphregularized energy function. While motivated differently, in this paper we carefully quantify explicit situations where the solutions they produce are equivalent and others where their properties sharply diverge. This includes the analysis of convergence, representational capacity, and interpretability. In support of this analysis, we also provide empirical head-to-head comparisons across multiple synthetic and public real-world node classification benchmarks.

show abstract

Deep Equilibrium Models

Cited by 76 publications

References 0 publications

Deep Equilibrium Models for Video Snapshot Compressive Imaging

Deep Equilibrium Models for Video Snapshot Compressive Imaging

Gradients are Not All You Need

Implicit vs Unfolded Graph Neural Networks

Contact Info

Product

Resources

About