Using a score to generalize the model performance into one numeric value has been one of the most popular approaches to empirically evaluate groundmotion models (GMMs). This approach has an advantage of simplifying model comparison. We study the effects of data correlation and score variability on the evaluation of GMMs. Most modern GMMs are hierarchical, in which ground motions from the same earthquake are modeled as correlated. We demonstrate, with examples, that incorrect results could occur if such hierarchical GMMs are evaluated by a score that does not duly address the data correlation. We propose to use the multivariate logarithmic score, a natural extension of the widely used univariate logarithmic score (referred to as LLH in the seismological literature), to correctly score hierarchical GMMs. The score variability affects the interpretation of model ranking. We demonstrate that the cluster bootstrap is a better bootstrap strategy, compared with other strategies proposed in the literature, to study the score variability. The bootstrap allows computing two useful quantities: the distinctness index that indicates if two models are truly different given the score variability and the frequency weight, a data-driven weighting scheme that represents the frequentist's interpretation of the weight of a logic-tree branch. The frequency weight has a direct link to the current practice of using multiple GMMs in a probabilistic seismic hazard assessment.Electronic Supplement: Python script to compute the multivariate logarithmic score for a hierarchical lognormal ground-motion model.

Modern, powerful techniques for the residual analysis of spatialtemporal point process models are reviewed and compared. These methods are applied to California earthquake forecast models used in the Collaboratory for the Study of Earthquake Predictability (CSEP). Assessments of these earthquake forecasting models have previously been performed using simple, low-power means such as the L-test and N-test. We instead propose residual methods based on rescaling, thinning, superposition, weighted K-functions and deviance residuals. Rescaled residuals can be useful for assessing the overall fit of a model, but as with thinning and superposition, rescaling is generally impractical when the conditional intensity λ is volatile. While residual thinning and superposition may be useful for identifying spatial locations where a model fits poorly, these methods have limited power when the modeled conditional intensity assumes extremely low or high values somewhere in the observation region, and this is commonly the case for earthquake forecasting models. A recently proposed hybrid method of thinning and superposition, called superthinning, is a more powerful alternative. The weighted K-function is powerful for evaluating the degree of clustering or inhibition in a model. Competing models are also compared using pixel-based approaches, such as Pearson residuals and deviance residuals. The different residual analysis techniques are demonstrated using the CSEP models and are used to highlight certain deficiencies in the models, such as the overprediction of seismicity in inter-fault zones for the model proposed by Helmstetter, Kagan and Jackson [Seismological

The engineering seismology community has recently recognized the importance of validating the performance of predictive models for seismic hazard by independent observations, yielding a number of studies on the relative performance of ground-motion prediction equations. The validation of intensity prediction equations (IPEs) has attracted less attention. We fill this gap by validating eight Italian IPEs plus one global IPE using five sets of Italian macroseismic intensity data, of which three are prospective and two retrospective to the models. We implemented multiple scoring methods to validate the models and found that the simple score of mean absolute error is sufficient to measure the general model performance. Good models consistently perform well under multiple methods and datasets, showing robustness. Models with physical functional forms are found to perform better. The global IPE performed well for Italian data, implying insignificant regional differences for IPEs. This result encourages grouping intensity data collected from multiple geographic regions, both from the Internet and traditional surveys, into a larger dataset for the use of future model development and validation.

Rescaling, thinning, and superposition are useful methods for the residual analysis of spatial‐temporal point processes. These techniques involve transforming the original point process into a new process that is a homogeneous Poisson process if and only if the fitted model is correct, so that one may inspect the residual process using standard tests for homogeneity as a means of assessing the goodness‐of‐fit of the model. Unfortunately, when the modeled conditional intensity of the original process is volatile, tests of homogeneity performed on residuals on the basis of these three residual methods tend to have low power. For such circumstances, we propose the method of super‐thinning, which combines thinned residuals and superposition. This technique involves the use of a tuning parameter, k, which controls how much thinning and superposition are performed to homogenize the process. The method is applied to the assessment of a parametric space–time point process model for the origin times and epicentral locations of recent major California earthquakes. Copyright © 2012 John Wiley & Sons, Ltd.

No abstract