Earthquake Likelihood Model Testing

Abstract

The Regional Earthquake Likelihood Models (RELM) project aims to produce and evaluate alternate models of earthquake potential (probability per unit volume, magnitude, and time) for California. Based on differing assumptions, these models are produced both to test the validity of their assumptions and explore which models should be incorporated in seismic hazard and risk evaluation. Tests based on physical and geological criteria are useful but here we focus on statistical methods using future earthquake data only. We envision two evaluations: a self-consistency test, and comparison of every pair of models for relative consistency. Both tests are based on the likelihood ratio method, and both would be fully prospective (that is, the models are not adjusted to fit the test data). To be tested, each model must assign a probability or probability density to any possible event within a specified region of space, time, and magnitude. For our tests the models must use a common format: earthquake rates in specified "bins" with location, magnitude, time and in some cases focal mechanism limits. IntroductionTo predict the behavior of a system is the desired proof of a model of this system. Seismology cannot predict earthquake occurence, however, it should seek for the best possible models to forecast earthquake occurence as precise as possible. This paper describes the rules of an experiment to examine or test earthquake forecasts in a statistical way. The primary purposes of the tests described below are to evaluate physical models for earthquakes, assure that source models used in seismic hazard and risk studies are consistent with earthquake data, and provide quantitative measures by which the models might be assigned weights in a future consensus model or be judged as suitable for particular areas.To test models against one another, we require that forecasts based on them can be expressed numerically in a standard format. That format is the average rate of earthquake occurrence within pre-specified limits of hypocentral latitude, longitude, magnitude, and time. For some source models there will also be 1

Results of the Regional Earthquake Likelihood Models (RELM) test of earthquake forecasts in California

“…It should be noted that the best average forecast λ in ¼ 2.84 × 10 −2 is one order of magnitude better than the random (no skill) forecast λ inr ¼ 2.86 × 10 −3 . A complex series of statistical tests based on maximum likelihood was proposed (18,19) to simultaneously evaluate both N e and λ i for each forecast. This approach was utilized to evaluate the forecasts after the first 2½ y of the 5-y test period (11,20).…”

A parimutuel gambling perspective to compare probabilistic seismicity forecasts

“…In many applications across the statistical sciences, likelihood ratios (or related information criteria, or Bayes factors) are used for pairwise model comparison. (For examples in the context of seismicity forecasting, we refer you to the articles by Schorlemmer et al 2007;Harte & Vere-Jones 2005;Marzocchi et al 2012.) The parimutuel gambling score can also be applied in pairwise fashion, and this permits a direct comparison of the resulting 'head-tohead' metric-a special case of parimutuel gambling with only two bettors-with the log-likelihood ratio.…”

Look-Ahead Benchmark Bias in Portfolio Performance Evaluation

“…But, one can never be 100% certain that all biases have been removed. In certain scientific fields concerned with forecasting, such as in earthquake prediction for instance, the community has recently evolved to the recognition that only real-time procedures can avoid such biases and test the validity of models (Jordan 2006, Schorlemmer et al 2007). Actually, real-time testing is a standard of the financial industry, since cautious investors only invest in funds that have developed a proven track record established over several years.…”