Merging expert and empirical data for rare event frequency estimation: Pool homogenisation for empirical Bayes models

Abstract

This version is available at https://strathprints.strath.ac.uk/27910/ Strathprints is designed to allow users to access the research output of the University of Strathclyde. Unless otherwise explicitly stated on the manuscript, Copyright © and Moral Rights for the papers on this site are retained by the individual authors and/or other copyright owners. Please check the manuscript for details of any other licences that may have been applied. You may not engage in further distribution of the material for any profitmaking activities or any commercial gain. You may freely distribute both the url (https://strathprints.strath.ac.uk/) and the content of this paper for research or private study, educational, or not-for-profit purposes without prior permission or charge.Any correspondence concerning this service should be sent to the AbstractEmpirical Bayes provides one approach to estimating the frequency of rare events as a weighted average of the frequencies of an event and a pool of events. The pool will draw upon, for example, events with similar precursors. The higher the degree of homogeneity of the pool, then the Empirical Bayes estimator will be more accurate. We propose and evaluate a new method using homogenisation factors under the assumption that events are generated from a HomogeneousPoisson Process. The homogenisation factors are scaling constants which can be elicited through structured expert judgement and used to align the frequencies of different events, hence homogenising the pool. The estimation error relative to the homogeneity of the pool is examined theoretically indicating that reduced error is associated with larger pool homogeneity. The effects of misspecified expert assessments of the homogenisation factors are examined theoretically and through simulation experiments. Our results show that the proposed Empirical Bayes method using homogenisation factors is robust under different degrees of misspecification.

A Bayes Linear Bayes Method for Estimation of Correlated Event Rates

“…We set . We assume the remaining homogenization factors are provided through expert judgment; see Quigley et al . for a further discussion on homogenization factors within HPPs.…”

“…We could estimate the parameters using maximum likelihood; however, this proved to be analytically intractable in the independent model of Quigley et al . Instead they used the method of moments . Although utilizing a numerical procedure to evaluate the maximum likelihood estimates is possible, the method of moments produces consistent estimates quickly and without computational burden.…”

“…Many approaches to the practical problems discussed, including the methods developed in Quigley et al ., assume an underlying model where the event rates of each process are assumed to be statistically independent, conditional on some unknowns. Within the context of Bayesian inference, if these unknowns are hyperparameters, which are specified exactly in the prior distribution, then the assumption of conditional statistical independence is equivalent to saying that making observations on one of the rates would not change our estimates, when updating, of any of the other rates.…”

“…An alternative to eliciting a prior is to use an empirical Bayes procedure where the events generated by each process are pooled to estimate each event rate as a weighted average of the pooled event rate and the observed frequency of that particular event from data. In Quigley et al ., homogenization factors are introduced to increase the effectiveness of the pooling process and hence the accuracy of the estimates obtained. To date, empirical Bayes inference for event rates has only been developed under an assumption of statistical independence.…”

A Shrinkage Approach for Failure Rate Estimation of Rare Events

“…They also provide an extensive case study of a railway application. The combination of Bayesian and EB has also been discussed . Sakalauskas considered the EB approach for the estimation of small rates with the Poisson–Gaussian model.…”