Adaptation of the Tuning Parameter in General Bayesian Inference with Robust Divergence

Abstract

We introduce a novel methodology for robust Bayesian estimation with robust divergence (e.g., density power divergence or γ-divergence), indexed by tuning parameters. It is well known that the posterior density induced by robust divergence gives highly robust estimators against outliers if the tuning parameter is appropriately and carefully chosen. In a Bayesian framework, one way to find the optimal tuning parameter would be using evidence (marginal likelihood). However, we theoretically and numerically illustrate that evidence induced by the density power divergence does not work to select the optimal tuning parameter since robust divergence is not regarded as a statistical model. To overcome the problems, we treat the exponential of robust divergence as an unnormalisable statistical model, and we estimate the tuning parameter via minimizing the Hyvarinen score. We also provide adaptive computational methods based on sequential Monte Carlo (SMC) samplers, which enables us to obtain the optimal tuning parameter and samples from posterior distributions simultaneously. The empirical performance of the proposed method through simulations and an application to real data is also provided.

General Bayesian Loss Function Selection and the use of Improper Models

“…There is also a growing literature following Basu et al (1998) that applies the Tsallis score to define a robust loss function to fit any given probability model. Such robustification depends on a hyper‐parameter that governs robustness‐efficiency trade‐offs and often leads to an improper model similar to Figure 1 (see Figure S1, and Yonekura and Sugasawa (2021) for a recent pre‐print building on our improper model interpretation to address the hyper‐parameter selection). In these scenarios traditional model selection tools are not applicable to choose the more appropriate loss.…”