Unbiased and efficient log-likelihood estimation with inverse binomial sampling

Abstract

The fate of scientific hypotheses often relies on the ability of a computational model to explain the data, quantified in modern statistical approaches by the likelihood function. The log-likelihood is the key element for parameter estimation and model evaluation. However, the log-likelihood of complex models in fields such as computational biology and neuroscience is often intractable to compute analytically or numerically. In those cases, researchers can often only estimate the log-likelihood by comparing observed data with synthetic observations generated by model simulations. Standard techniques to approximate the likelihood via simulation either use summary statistics of the data or are at risk of producing substantial biases in the estimate. Here, we explore another method, inverse binomial sampling (IBS), which can estimate the log-likelihood of an entire data set efficiently and without bias. For each observation, IBS draws samples from the simulator model until one matches the observation. The log-likelihood estimate is then a function of the number of samples drawn. The variance of this estimator is uniformly bounded, achieves the minimum variance for an unbiased estimator, and we can compute calibrated estimates of the variance. We provide theoretical arguments in favor of IBS and an empirical assessment of the method for maximum-likelihood estimation with simulation-based models. As case studies, we take three model-fitting problems of increasing complexity from computational and cognitive neuroscience. In all problems, IBS generally produces lower error in the estimated parameters and maximum log-likelihood values than alternative sampling methods with the same average number of samples. Our results demonstrate the potential of IBS as a practical, robust, and easy to implement method for log-likelihood evaluation when exact techniques are not available.

Uncertainty is maintained and used in working memory

“…Additionally, computational models could be used to decode uncertainty from neural activity in working memory tasks. Work in visual perception demonstrates that uncertainty information is represented in primary visual cortex (van Bergen et al, 2015;van Bergen, 2019;Walker et al, 2020;Hénaff et al, 2020). These studies built normative Bayesian models to infer stimulus value from functional magnetic resonance imaging blood oxygen level dependent (BOLD) signal.…”

“…For each participant, we used maximum-likelihood estimation to find which parameter combination best describes participants’ data. Computing the LL analytically is intractable, so we used inverse binomial sampling (IBS; van Opheusden et al., 2020 ), a method that efficiently computes an unbiased estimate of the LL. This calculation is stochastic, so we additionally used an optimization algorithm, Bayesian Adaptive Direct Search (BADS), that can account for stochasticity and expensive LL evaluations ( Acerbi & Ma, 2017 ).…”

Measuring the accuracy of likelihood-free inference

“…Part of IPM, explicitly depends on function of interest [2,17,27] Integral probability metrics [33] Standard class of metrics on probability distributions (TV) [20] (MMD) [25] (CDF) [39] Acceptance rate Easy to check, sample size for rejection sampling [6,7,12,18,29,32,31,34,35] Effective sample size [9,19] Adjusting acceptance rate for sample weights [10,23,28,29,30] Normalized posterior MSE Natural metric on distributions [4,5] Unnormalized posterior MSE Normalization difficult to analyze [16] φ-divergences [33] Standard divergences between probability distributions (KL) [3,12,18,24], (TV) [20] Log-likelihood accuracy Penalizes misestimating likelihood as zero [37] Null hypothesis significance tests…”

Uncertainty is Maintained and Used in Working Memory

“…For each participant, we used maximum-likelihood estimation to find which parameter combination best describes participant’s data. Computing the LL analytically is intractable, so we used Inverse Binomial Sampling (IBS; van Opheusden, Acerbi, & Ma, 2020), a method which efficiently computes an unbiased estimate of the LL. This calculation is stochastic, so we additionally used an optimization algorithm that can account for stochasticity and expensive LL evaluations (BADS; Acerbi & Ma, 2017).…”