Gradients are Not All You Need

Abstract

Differentiable programming techniques are widely used in the community and are responsible for the machine learning renaissance of the past several decades. While these methods are powerful, they have limits. In this short report, we discuss a common chaos based failure mode which appears in a variety of differentiable circumstances, ranging from recurrent neural networks and numerical physics simulation to training learned optimizers. We trace this failure to the spectrum of the Jacobian of the system under study, and provide criteria for when a practitioner might expect this failure to spoil their differentiation based optimization algorithms. * Equal contribution 2 Up to numerical precision, though see [Chow and Palmer, 1992, Kachman et al., 2017] for cases where the gradients will be close.Preprint. Under review.

Forecasting Global Weather with Graph Neural Networks

“…We used a progressively larger rollout for each round of training: 4, 8, and 12-step losses, corresponding to 1, 2, and 3-day rollouts, for the three rounds of training. Using even larger rollouts is enticing, but there are probably diminishing returns [Metz et al, 2021], and in practice we obtained only slightly worse results when using a 4-step loss throughout.…”

“…As an aside we note that, while it may be tempting to replace this heuristic with a a more end-to-end-learned approach, (i) you would still have to use human judgement to pick a metric to optimize (e.g. globe-averaged Z500 at a 10-day forecast horizon) and (ii) directly optimizing over tens of rollout steps might not be effective [Metz et al, 2021], even if you are able to fit the gradient into GPU memory.…”

Analytically Integratable Zero-restlength Springs for Capturing Dynamic Modes unrepresented by Quasistatic Neural Networks

“…We quite robustly addressed this by isolating nondimensionalized functions that were trivially carefully implemented to obtain the correct asymptotic result in all cases. For more discussions on both numerical and analytical issues with gradients, we refer the interested readers to [Johnson and Fedkiw 2022;Metz et al 2021].…”

EvoJAX: Hardware-Accelerated Neuroevolution

“…But much of this progress is restricted to systems that rely on gradient descent, a highly effective optimization method when we provide it with a well-defined, differentiable objective function. But in areas such as artificial life, complex systems, computational biology, and even classical physics [17], much of the interesting behaviors we observe take place near the chaotic states, where a system is constantly transitioning between order and disorder. It can be argued that intelligence life and even civilization are all complex systems operating at the edge of chaos [3,15].…”