Prediction uncertainty and optimal experimental design for learning dynamical systems

Abstract

Dynamical systems are frequently used to model biological systems. When these models are fit to data, it is necessary to ascertain the uncertainty in the model fit. Here, we present prediction deviation, a metric of uncertainty that determines the extent to which observed data have constrained the model's predictions. This is accomplished by solving an optimization problem that searches for a pair of models that each provides a good fit for the observed data, yet has maximally different predictions. We develop a method for estimating a priori the impact that additional experiments would have on the prediction deviation, allowing the experimenter to design a set of experiments that would most reduce uncertainty. We use prediction deviation to assess uncertainty in a model of interferon-alpha inhibition of viral infection, and to select a sequence of experiments that reduces this uncertainty. Finally, we prove a theoretical result which shows that prediction deviation provides bounds on the trajectories of the underlying true model. These results show that prediction deviation is a meaningful metric of uncertainty that can be used for optimal experimental design.

Estimating parametric phenotypes that determine anthesis date in Zea mays: Challenges in combining ecophysiological models with genetics

“…However, given the expense of such large-scale trials and the multiple purposes they serve, “balance” cannot mean “orthogonality” with all lines planted at all sites. There is a large literature on methods for optimizing experimental designs [ 52 – 54 ]. Perhaps such methods should be applied at levels higher than the single field trial with the needs of GSP estimation being a specific criterion receiving consideration.…”

Differential equations in data analysis

“…Liepe, Filippi, Komorowski, and Stumpf (2013) focused on an experimental design that attempts to maximize information regarding model parameters. Letham, Letham, Rudin, and Browne (2016) developed a method that uses prediction as a measure for testing the usefulness of carrying out additional experiments. They derived theoretical properties of their method (note the small correction presented in Letham, Letham, Rudin, and Browne (2017)) and applied it to real data.…”

Variational analysis of inference from dynamical systems

“…fitting the values of the process with models in D. This type of fitting occurs frequently in fields such as systems biology and ecology, see for example [6,35,36,41,55].…”

“…In several applied fields, there is interest in fitting parametrized families of dynamical systems to observations. Indeed, there are examples in ecology [36,55], geophysical modeling [3,22], systems biology [6,35,41], and data assimilation [34]. As explained in greater detail in [39], the variational approach taken here may be useful in analyzing the fitting methods in settings such as these.…”