Using Machine Learning to Anticipate Tipping Points and Extrapolate to Post-Tipping Dynamics of Non-Stationary Dynamical Systems

Abstract

In this paper we consider the machine learning (ML) task of predicting tipping point transitions and long-term post-tipping-point behavior associated with the time evolution of an unknown (or partially unknown), non-stationary, potentially noisy and chaotic, dynamical system. We focus on the particularly challenging situation where the past dynamical state time series that is available for ML training predominantly lies in a restricted region of the state space, while the behavior to be predicted evolves on a larger state space set not fully observed by the ML model during training. In this situation, it is required that the ML prediction system have the ability to extrapolate to different dynamics past that which is observed during training. We investigate the extent to which ML methods are capable of accomplishing useful results for this task, as well as conditions under which they fail. In general, we found that the ML methods were surprisingly effective even in situations that were extremely challenging, but do (as one would expect) fail when "too much" extrapolation is required. For the latter case, we investigate the effectiveness of combining the ML approach with conventional modeling based on scientific knowledge, thus forming a hybrid prediction system which we find can enable useful prediction even when its ML-based and knowledge-based components fail when acting alone. We also found that achieving useful results may require using very carefully selected ML hyperparameters and we propose a hyperparameter optimization strategy to address this problem. The main conclusion of this paper is that ML-based approaches are promising tools for predicting the behavior of non-stationary dynamical systems even in the case where the future evolution (perhaps due to the crossing of a tipping point) includes dynamics on a set outside of that explored by the training data.

Reconstructing Computational Dynamics from Neural Measurements with Recurrent Neural Networks

“…In DS reconstruction, it is therefore important to check for geometrical and other time-invariant properties of the systems under study. For instance, measures based on the Kullback-Leibler divergence (Brenner et al, 2022; Koppe et al, 2019) or Wasserstein distance (Patel & Ott, 2022) have been used to assess the geometrical overlap in data point distributions in the large time limit across state spaces generated by the true and the reconstructed system. The maximal Lyapunov exponent or the so-called correlation dimension, an empirical estimate of the fractal dimensionality of an attractor, are other examples of such invariant measures (Kantz & Schreiber, 2004; see also Gilpin, 2020).…”

“…However, as far as DS reconstruction is concerned, this topic has not been much explored yet. It is as yet unclear whether and how DS can be retrieved from time series measurements which entail slow parameter drifts (Patel & Ott, 2022). The problem here is partly related to the exploding/vanishing gradient issue briefly raised in sect.…”

“…Finally, there is the issue of generalization : If our dynamic model is correct, it should be able to generalize beyond the data domain seen in training, sometimes labeled ‘out-of-distribution’ generalization in machine learning (in contrast to ‘mere’ out-of-sample prediction). The problem of drifting parameters considered above may be seen as one facet of this issue, as a good model should be able to predict dynamics beyond the bifurcation point within unseen parameter regimes (Patel & Ott, 2022). Experimental manipulations could be designed to test such out-of-distribution predictions: For instance, if we have a representation of different neuron types or cortical layers in our model (as in Fig.…”