On the Koopman Operator of Algorithms

Dietrich, Felix; Thiem, Thomas N.; Kevrekidis, Ioannis G.

doi:10.1137/19m1277059

Cited by 27 publications

(21 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The main idea is that the iterates of the algorithm, in the high-dimensional ambient space naturally "concentrate" (due to scale separation of time scales, eigenvalues, sensitivities) on lower-dimensional manifolds (e.g. see [47]). The common language of conditional KL expansions can then seamlessly (a) deduce the parametrization of a useful reduced "latent space"; (b) provide a local surrogate (what we would like to call a "targeted surrogate" in the sense that it is not global, but rather "just enough" for the next algorithm iteration) in this latent space that (c) can be used to design the next algorithm iteration in this targeted latent space; and then also (d) translate the results to the full space ("lifting"), where the full model will be briefly used (to briefly simulate, or to evaluate the expensive objective function).…”

Section: Discussion and Outlookmentioning

confidence: 99%

On the Correspondence between Gaussian Processes and Geometric Harmonics

Dietrich,

Bello-Rivas,

Kevrekidis

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We discuss the correspondence between Gaussian process regression and Geometric Harmonics, two similar kernel-based methods that are typically used in different contexts. Research communities surrounding the two concepts often pursue different goals. Results from both camps can be successfully combined, providing alternative interpretations of uncertainty in terms of error estimation, or leading towards accelerated Bayesian Optimization due to dimensionality reduction.

show abstract

Section: Discussion and Outlookmentioning

confidence: 99%

On the Correspondence between Gaussian Processes and Geometric Harmonics

Dietrich,

Bello-Rivas,

Kevrekidis

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Finally, we see this work as being part of a growing body of literature that is bringing attention to the fact that dynamical systems theory, and in particular KOT, can be used for problems that have historically relied on optimization theory [12][13][14]64 . These papers have highlighted the fact that, while optimization theory has its advantages, its de-emphasis on the past history of the system for computing the future state (e.g.…”

Section: Discussionmentioning

confidence: 99%

On Koopman Mode Decomposition and Tensor Component Analysis

Redman

2021

Preprint

View full text Add to dashboard Cite

Koopman mode decomposition and tensor component analysis are two tools that decompose high dimensional data sets into low dimensional modes that capture relevant features and/or dynamics. Despite their similar goal, the two methods are largely used by distinct scientific communities. For the first time, examine the two together and show that, under a certain (reasonable) condition on the data, the theoretical decomposition given by tensor component analysis is the same as that given by Koopman mode decomposition. This realization provides possibilities for new algorithmic approaches to both Koopman mode decomposition and tensor component analysis, new insight into what is being captured by the tensor component analysis modes, and a "bridge" with which the two communities can more effectively communicate.Koopman operator theory (KOT) has emerged as a powerful and general framework with which to understand nonlinear dynamical systems in a data-driven manner. Despite its many strengths, there are numerous scientific fields, such as neuroscience and biology, where it remains under employed. As part of this stems from KOT being formulated mathematically in a distinct way from existing, commonly used methods, such as principal component analysis (PCA), there is a need for bridging KOT with such methods. Here, for the first time, we investigate how a major KOT analysis method differs from tensor component analysis (TCA), an extension of PCA that has quickly become adopted by some of those same fields hesitant on KOT. We show that, in certain scenarios, TCA and KOT, in theory, give the same decomposition of data. This not only makes strides in establishing a bridge between TCA and KOT, but also provides new insight into TCA and new possible algorithms for implementing KOT and TCA.

show abstract

“…Another author took a similar perspective to identify KOs of interest for NN training [10]. Unbeknownst (and in parallel) to us, general work connecting KOT to algorithms (including GD) was very recently explored and offered as a way in which NN training could be sped up [15]. However, while [15] focused on solving numerical problems by constructing the KO associated with GD, we provide a full fledged study of NN training that uses GD.…”

Section: Related Workmentioning

confidence: 99%

Optimizing Neural Networks via Koopman Operator Theory

Dogra¹,

Redman²

2020

Preprint

View full text Add to dashboard Cite

Koopman operator theory, a powerful framework for discovering the underlying dynamics of nonlinear dynamical systems, was recently shown to be intimately connected with neural network training. In this work, we take the first steps in making use of this connection. As Koopman operator theory is a linear theory, a successful implementation of it in evolving network weights and biases offers the promise of accelerated training, especially in the context of deep networks, where optimization is inherently a non-convex problem. We show that Koopman operator theory methods allow for accurate predictions of the weights and biases of a feedforward, fully connected deep network over a non-trivial range of training time. During this time window, we find that our approach is at least 10x faster than gradient descent based methods, in line with the results expected from our complexity analysis. We highlight additional methods by which our results can be expanded to broader classes of networks and larger time intervals, which shall be the focus of future work in this novel intersection between dynamical systems and neural network theory. IntroductionDespite their black box nature, the training of artificial neural networks (NNs) is a discrete dynamical system. During training, NN weights evolve along a trajectory in an abstract weight space, the path determined by the implemented learning algorithm, the data used for training, and the network architecture. This dynamical systems picture is familiar, as many introductions to learning algorithms, such as gradient descent (GD), attempt to visualize training as a process whereby weights are changed iteratively under the influence of the loss landscape. Yet, while dynamical systems theory has provided insight into the behavior of many complex systems, its application to NNs has been limited.Recent advances in Koopman operator theory (KOT) have made it a powerful tool in studying the underlying dynamics of nonlinear systems in a data-driven manner [1][2][3][4][5][6][7][8][9]. This begs the question, can KOT be used to learn and predict the dynamics present in NN training? If so, can such an approach, which we call Koopman training, afford us benefits that traditional NN training methods cannot? * The authors contributed equally Preprint. Under review.

show abstract

On the Koopman Operator of Algorithms

Cited by 27 publications

References 38 publications

On the Correspondence between Gaussian Processes and Geometric Harmonics

On the Correspondence between Gaussian Processes and Geometric Harmonics

On Koopman Mode Decomposition and Tensor Component Analysis

Optimizing Neural Networks via Koopman Operator Theory

Contact Info

Product

Resources

About