Closed-form Continuous-time Neural Models

Hasani, Ramin; Lechner, Mathias; Amini, Alexander; Liebenwein, Lucas; Ray, Aaron; Tschaikowski, Max; Rus, Daniela

doi:10.48550/arxiv.2106.13898

Cited by 6 publications

(8 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although GoTube is considerably more computationally efficient than existing methods, the dimensionality of the system-under-test as well as the type of numerical ODE solver exponentially affect their performance. We can improve on this limitation by using Hypersolvers (Poli et al, 2020), closed-form continuous depth models (Hasani et al, 2021a), and compressed representations of neural ODEs (Liebenwein et al, 2021).…”

Section: Discussion Scope and Conclusionmentioning

confidence: 99%

GoTube: Scalable Stochastic Verification of Continuous-Depth Models

Gruenbacher¹,

Lechner²,

Hasani³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We introduce a new stochastic verification algorithm that formally quantifies the behavioural robustness of any time-continuous process formulated as a continuousdepth model. The algorithm solves a set of global optimization (Go) problems over a given time horizon to construct a tight enclosure (Tube) of the set of all process executions starting from a ball of initial states. We call our algorithm GoTube. Through its construction, GoTube ensures that the bounding tube is conservative up to a desired probability. GoTube is implemented in JAX and optimized to scale to complex continuous-depth models. Compared to advanced reachability analysis tools for time-continuous neural networks, GoTube provably does not accumulate over-approximation errors between time steps, and avoids the infamous wrapping effect inherent in symbolic techniques. We show that GoTube substantially outperforms state-of-the-art verification tools in terms of the size of the initial ball, speed, time-horizon, task completion, and scalability, on a large set of experiments. GoTube is stable and sets the state-of-the-art for its ability to scale up to time-horizons well-beyond what has been possible before.

show abstract

Section: Discussion Scope and Conclusionmentioning

confidence: 99%

GoTube: Scalable Stochastic Verification of Continuous-Depth Models

Gruenbacher¹,

Lechner²,

Hasani³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Other approaches include directly predicting the resulting value at a variable time (Mattheakis, Joy, and Protopapas 2021;Chen et al 2020a) and architectures with closed-form time propagation (Hasani et al 2021). We take the approach of augmenting a traditional solver, but only focus on Hamiltonian systems.…”

Section: Related Workmentioning

confidence: 99%

Hyperverlet: A Symplectic Hypersolver for Hamiltonian Systems

Mathiesen

Yang

2022

AAAI

View full text Add to dashboard Cite

Hamiltonian systems represent an important class of dynamical systems such as pendulums, molecular dynamics, and cosmic systems. The choice of solvers is significant to the accuracy when simulating Hamiltonian systems, where symplectic solvers show great significance. Recent advances in neural network-based hypersolvers, though achieve competitive results, still lack the symplecity necessary for reliable simulations, especially over long time horizons. To alleviate this, we introduce Hyperverlet, a new hypersolver composing the traditional, symplectic velocity Verlet and symplectic neural network-based solvers. More specifically, we propose a parameterization of symplectic neural networks and prove that hyperbolic tangent is r-finite expanding the set of allowable activation functions for symplectic neural networks, improving the accuracy. Extensive experiments on a spring-mass and a pendulum system justify the design choices and suggest that Hyperverlet outperforms both traditional solvers and hypersolvers.

show abstract

“…This category includes the UnICORNN [47] and its predecessor coRNN [46] which discretize a second-order ODE inspired by oscillatory systems. Other models include the Liquid Time-Constant Networks (LTC) [27] and successor CfC [26], which use underlying dynamical systems with varying time-constants with stable behavior and provable rates of expressivity measured by trajectory length. The LTC is based on earlier dynamic causal models (DCM) [21], which are a particular ODE related to state spaces with an extra bilinear term.…”

Section: A Related Workmentioning

confidence: 99%

Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers

Gu¹,

Johnson²,

Goel³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recurrent neural networks (RNNs), temporal convolutions, and neural differential equations (NDEs) are popular families of deep learning models for time-series data, each with unique strengths and tradeoffs in modeling power and computational efficiency. We introduce a simple sequence model inspired by control systems that generalizes these approaches while addressing their shortcomings. The Linear State-Space Layer (LSSL) maps a sequence u → y by simply simulating a linear continuous-time state-space representation ẋ = Ax + Bu, y = Cx + Du. Theoretically, we show that LSSL models are closely related to the three aforementioned families of models and inherit their strengths. For example, they generalize convolutions to continuous-time, explain common RNN heuristics, and share features of NDEs such as time-scale adaptation. We then incorporate and generalize recent theory on continuous-time memorization to introduce a trainable subset of structured matrices A that endow LSSLs with long-range memory. Empirically, stacking LSSL layers into a simple deep neural network obtains state-of-the-art results across time series benchmarks for long dependencies in sequential image classification, real-world healthcare regression tasks, and speech. On a difficult speech classification task with length-16000 sequences, LSSL outperforms prior approaches by 24 accuracy points, and even outperforms baselines that use hand-crafted features on 100x shorter sequences.

show abstract

Closed-form Continuous-time Neural Models

Cited by 6 publications

References 31 publications

GoTube: Scalable Stochastic Verification of Continuous-Depth Models

GoTube: Scalable Stochastic Verification of Continuous-Depth Models

Hyperverlet: A Symplectic Hypersolver for Hamiltonian Systems

Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers

Contact Info

Product

Resources

About