Convergence properties of gradient descent noise reduction

Abstract

Gradient descent noise reduction is a technique that attempts to recover the true signal, or trajectory, from noisy observations of a non-linear dynamical system for which the dynamics are known. This paper provides the first rigorous proof that the algorithm will recover the original trajectory for a broad class of dynamical systems under certain conditions. The proof is obtained using ideas from linearisation theory. Since the first introduction of the algorithm it has been recognised that the algorithm can fail to recover the true trajectory, and it has been suggested that this is a practical or numerical limitation that is a consequence of near tangencies between stable and unstable manifolds. This paper demonstrates through numerical experiments and details of the proof that the situation is worse than expected in that near tangencies impose essential limitations on noise reduction, not just practical or numerical limitations. That is, gradient descent noise reduction will sometimes fail to recover the true trajectory, even with unlimited, perfect computation. On the other hand, the numerical experiments suggest that the gradient descent noise reduction algorithm will always recover a trajectory that is entirely consistent with the evidence provided by the observations, that is, it attains the best that can be achieved given the observations. It is argued that near tangencies will therefore impose the same limitations on any noise reduction algorithm.

On variational data assimilation in continuous time

“…In Bröcker and Parlitz (2001) and Ridout and Judd (2002), a very similar problem is discussed, although the approach is different from Farmer and Sidorovich (1990). Both papers propose to minimize the indeterminism…”

“…Nonetheless, the approach appears to work well, at least if model error is absent. Using the methodology presented in this paper, an interpretation of this somewhat curious finding of Ridout and Judd (2002) and Bröcker and Parlitz (2001) is hazarded. It seems that the solution strategy corresponds to a (somewhat crude) continuation scheme.…”

“…The solution for small α is {x, λ} = {η, 0}, exactly what is used by Bröcker and Parlitz (2001) and Ridout and Judd (2002) to initialize their algorithms. We can conclude that the solution strategy in these publications corresponds to a continuation scheme in which α is moved from 0 to 1, which is appropriate for small dynamical errors.…”

“…Le Dimet and Talagrand, 1986;Talagrand and Courtier, 1987;Courtier and Talagrand, 1987;Rabier et al, 1993;Pires et al, 1996;Apte et al, 2008, a list which is by no means exhaustive). In Farmer and Sidorovich (1990), Bröcker and Parlitz (2001), and Ridout and Judd (2002), a very similar approach was considered (at least as a motivation), albeit from a nonlinear dynamics perspective; furthermore, different solution strategies were employed. Several interesting observations were reported.…”

Shadowing Pseudo-Orbits and Gradient Descent Noise Reduction

“…There have been developed algorithms for approximating shadowing trajectories, most notably gradient descent methods [4,7,12,13]. Recent proofs show that these methods converge to the true trajectory of a hyperbolic system, given observations with sufficiently small noise and a perfect model [4,28]. When noise levels are larger, or the system is non-hyperbolic, then it has been described how near tangencies of invariant stable and unstable manifolds can cause the failure of numerical methods for finding shadowing trajectories [28].…”