Computer model validation with functional output

Abstract

A key question in evaluation of computer models is Does the computer model adequately represent reality? A six-step process for computer model validation is set out in Bayarri et al. [Technometrics 49 (2007) 138--154] (and briefly summarized below), based on comparison of computer model runs with field data of the process being modeled. The methodology is particularly suited to treating the major issues associated with the validation process: quantifying multiple sources of error and uncertainty in computer models; combining multiple sources of information; and being able to adapt to different, but related scenarios. Two complications that frequently arise in practice are the need to deal with highly irregular functional data and the need to acknowledge and incorporate uncertainty in the inputs. We develop methodology to deal with both complications. A key part of the approach utilizes a wavelet representation of the functional data, applies a hierarchical version of the scalar validation methodology to the wavelet coefficients, and transforms back, to ultimately compare computer model output with field output. The generality of the methodology is only limited by the capability of a combination of computational tools and the appropriateness of decompositions of the sort (wavelets) employed here. The methods and analyses we present are illustrated with a test bed dynamic stress analysis for a particular engineering system.Comment: Published in at http://dx.doi.org/10.1214/009053607000000163 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org

Statistical Emulation of Climate Model Projections Based on Precomputed GCM Runs*

“…The GP approach to computer model emulation assumes that the output of interest is a Gaussian process in some set of inputs that vary across model runs. Among others, Challenor et al (2010) and Rougier (2008) have discussed extensions to the GP approach to multivariate climate output, and several authors have proposed approaches for multivariate, time-dependent output: projection on a lower dimensional space via principal component analysis (Wilkinson 2010;Higdon et al 2008) or wavelet decomposition (Bayarri et al 2007), choice of a single representative output (Challenor et al 2006) or a spatial aggregated average of it (Hankin 2005), kernel mixing and matrix identities (Sham Bhat et al 2012), and dynamically autoregressive models (Fei and West 2009).…”

Choosing the Sample Size of a Computer Experiment: A Practical Guide

“…Although the output from the computer model is often multivariate, we will restrict our attention to scalar output. The results for scalar output can be carried over by using principal component analysis or wavelet decompositions of functional output as in Higdon et al (2005) and Bayarri et al (2007).…”

Bayesian calibration of the Thermosphere-Ionosphere Electrodynamics General Circulation Model (TIE-GCM)

“…Since the outputs of TIE-GCM are effectively continuous (though discretized) quantities distributed in space and time, to carry out the calibration, we could have followed recent functional approaches (Bayarri et al, 2007;Higdon et al, 2008) by decomposing in wavelets bases or according to the first few principal components. We could have used periodic Fourier bases as we did for the direct emulation (Rougier, 2008), since they worked well there.…”