Optimal experimental design for machine learning using the Fisher information matrix

Abstract

Optimal experimental design (OED) refers to a class of methods for selecting new data collection conditions that minimize the statistical uncertainty in the inferred parameter values of a model. The Fisher information matrix (FIM) gives an estimate of the relative uncertainty in and correlation among the model parameters based on the local curvature of the cost function. FIM-based approaches to OED allow for rapid assessment of many different experimental conditions (e.g., input data type, parameterizations, etc.). In machine learning models, accurate parameter estimates are often not a priority (nor even desirable) as they have no direct physical meaning. Instead, one would like to minimize the uncertainty in the model predictions for several quantities of interest. FIM approaches to OED can be generalized to minimize statistical variance, not in parameters, but in predictions of the quantities of interest. This approach has been applied, for example, to systems biology models of biochemical reaction networks [Transtrum and Qiu, BMC Bioinformatics 13(1), 181 (2012)]. Preliminary application of the FIM to optimize experimental design for source localization in an uncertain ocean environment is a first step towards an efficient machine learning algorithm that produces results with the least uncertainty in the quantities of interest.

An optimal calibration method for gyro and star sensor based on observability analysis

“…According to the theory of the Cramer-Rao lower bound (CRLB), 30 there is the following relationship between the estimation error covariance P and FIM…”

On Optimal Test Signal Design and Parameter Identification Schemes for Dynamic Takagi-Sugeno Fuzzy Models Using the Fisher Information Matrix

“…[21] circumvents the calculation of the posterior by representing the desired samples as a reference trajectory, such that the input design task is transformed into an optimal control problem. Neilsen [22] uses the FIM to assess the uncertainty of other quantities than the model parameters that are more important to the applications dealt with. Ples ¸u [23] uses the FIM to assess the global sensitivity before any experimental design because the importance of high-quality initial models is emphasized.…”

Seabed classification using physics-based modeling and machine learning

“…As will be discussed later, MFI can still be employed towards this task, but, here, we are mostly interested in investigating the potential of machine learning in sediment classification. Recently, a sensitivity analysis was performed on parameters in a two layer seafloor, indicating the promise of these methods [47]. We investigate a low frequency case with the sound being transmitted by a source and received at a vertical line array with 20 hydrophones (see Figure 1).…”