Information geometry for multiparameter models: new perspectives on the origin of simplicity

Abstract

Complex models in physics, biology, economics, and engineering are often ill-determined or sloppy: their multiple parameters can vary over wide ranges without significant changes in their predictions. This review uses the tools of information geometry to explore this phenomenon, and the deep relations between sloppiness and emergent theories. We introduce information geometry, the model manifold of predictions whose coordinates are the model parameters, and its hyperribbon structure. These hyperribbons explain why only a few parameter combinations have significant effects on the behavior of complex multiparameter models. We review recent rigorous results using approximation theory to connect the hierarchy of hyperribbon widths to approximation theory, and to the smoothness of model predictions under changes of the system control variables. We discuss recent methods for following sloppy geodesics to find simpler models on the boundary of the model manifold -- emergent theories with fewer parameters that explain the behavior equally well. We discuss a recent `optimal' Bayesian prior which maximizes the mutual information between model parameters and experimental data. This prior naturally favors points on the emergent boundary theories, thus selecting simpler models when they make nearly equivalent predictions. We introduce a `projected maximum likelihood' prior that efficiently approximates this optimal prior, and contrast both to the poor behavior of the traditional Jeffrey's prior. We discuss the way the renormalization group coarse-graining in statistical mechanics introduces a flow of the model manifold, and connect stiff and sloppy directions along the model manifold with relevant and irrelevant eigendirections of the renormalization group. Finally, we discuss recently developed `intensive' embedding methods, allowing one to visualize the predictions of arbitrary probabilistic models as low-dimensional projections of an isometric embedding, and illustrate our method by generating the model manifold of the Ising model.

How Occam’s razor guides human decision-making

“…Another potentially intriguing future direction is a comparison with other formal approaches to the emergence of simplicity that can lead to different predictions. Recent studies have argued that Jeffrey’s prior (upon which our geometric approach is based) could give an incomplete picture of the complexity of a class of models that occur commonly in the natural sciences, which contain many combinations of parameters that do not affect model behavior, and proposed instead the use of data-dependent priors ( 43 , 44 ). The two methods lead to different results, especially in the data-limited regime ( 45 ).…”

Far from Asymptopia: Unbiased High-Dimensional Inference Cannot Assume Unlimited Data

“…While our concern here is with the ideal properties, for practical use nearly optimal approximations may be required. One possibility is the adaptive slab-and-spike prior introduced in [ 6 ]. Another would be to use some variational family with adjustable meta-parameters [ 41 ].…”

“…This is the approach of [ 41 ], and of many papers maximizing other scores, often described as “variational”. Another prior that typically has large was introduced in [ 6 ] under the name “adaptive slab-and-spike prior”. It pulls every point x in a distribution back to its maximum likelihood point : The result has weight everywhere in the model manifold, but has extra weight on the edges.…”

“…This may be thought of as having units of standard deviations, so that along a line of integrated length L there are about L distinguishable points, and thus any parameter which can be measured to a few digits of precision has length . It is a striking empirical feature of models in science that most have a few such long (or relevant) parameter directions, followed by many more short (or irrelevant) orthogonal directions [ 3 , 4 , 5 , 6 ]. The irrelevant lengths, all , show a characteristic spectrum of being roughly evenly spaced on a log scale, often over many decades.…”

Sloppiness: Fundamental study, new formalism and its application in model assessment

“…Further, they show that specific models are not sloppy at microscopic scales, and sloppiness emerges only when fit to collective behaviour. Transtrum et al and Katherine et al suggest that the existence of sloppiness (dependence of model parameters to only a few macroscopic parameter directions) is the origin of simplicity in science and responsible for the emergence of comprehensible macroscopic theories from highly complex microscopic processes and [34,35]. Evangelou et al have developed a method to identify important and unimportant parameter combinations using manifold learning techniques (Dmaps).…”