Designing accurate emulators for scientific processes using calibration-driven deep models

Abstract

Predictive models that accurately emulate complex scientific processes can achieve speed-ups over numerical simulators or experiments and at the same time provide surrogates for improving the subsequent analysis. Consequently, there is a recent surge in utilizing modern machine learning methods to build data-driven emulators. In this work, we study an often overlooked, yet important, problem of choosing loss functions while designing such emulators. Popular choices such as the mean squared error or the mean absolute error are based on a symmetric noise assumption and can be unsuitable for heterogeneous data or asymmetric noise distributions. We propose Learn-by-Calibrating, a novel deep learning approach based on interval calibration for designing emulators that can effectively recover the inherent noise structure without any explicit priors. Using a large suite of use-cases, we demonstrate the efficacy of our approach in providing high-quality emulators, when compared to widely-adopted loss function choices, even in small-data regimes.

Training Calibration-based Counterfactual Explainers for Deep Learning Models in Medical Image Analysis

“…Predictive Model Design using LbC. While conventional metrics such as cross entropy (for categorical-valued outputs) and mean squared error (for continuous-valued outputs) are commonly used, it has been recently found that interval calibration is effective for obtaining accurate and well-calibrated predictive models 28 . Hence, in TraCE, we adapt the Learn-by-Calibrating approach to train classifier (or regression) models that map from the CXR latent space to a desired target variable.…”

“…), the intervals are considered to be well-calibrated if the likelihood matches the confidence level. Denoting the parameters of the models F and G by θ and φ respectively, we use an alternating optimization strategy similar to 28 . In order to update φ , we use the empirical interval calibration error as the objective:…”

“…For example, uncertainties from Monte-Carlo dropout 46 and direct error prediction 47 have been used to perform confidence calibration in deep classifiers. Similarly, the recently proposed Learn-by-Calibrating (LbC) approach 28 introduced an interval calibration objective based on uncertainty estimates for training deep regression models.…”

“…along with its prediction uncertainty; and (iii) a counterfactual optimization strategy that uses an uncertainty-based calibration objective to reliably elucidate the intricate relationships between image signatures and the target attribute. While our approach is flexible to support the use of any uncertainty estimator or prediction models that use explicit regularization to produce well-calibrated predictions, TraCE builds upon the recent Learn-by-Calibrating (LbC) technique 28 to obtain prediction intervals for both classification and regression settings. LbC jointly trains an auxiliary interval estimator alongside the predictor model using an interval calibration objective, and has been shown to be effective at recovering complex function mappings in scientific datasets.…”

The data-driven future of high-energy-density physics

“…Work to optimize the three information sources in Figure 3 include the acceleration of multiphysics simulations using deep learning 55,97 , infrastructure for intelligent control of large-scale simulations 98 , intelligent and data-informed design of experiments 42,43,99 , as well as optimization of experimental facilities 100 . These developments have enabled simulation studies of unprecedented size 101 and the generation of open-source ICF datasets 102 that motivate novel deep learning research 54,57,103,104 . AI tools have been applied to the automatic analysis and featurization of complex data types like spectra, images 54,105 , and line-of-sight dependent quantities.…”