Comprehensive benchmarking of Markov chain Monte Carlo methods for dynamical systems

Abstract

BackgroundIn quantitative biology, mathematical models are used to describe and analyze biological processes. The parameters of these models are usually unknown and need to be estimated from experimental data using statistical methods. In particular, Markov chain Monte Carlo (MCMC) methods have become increasingly popular as they allow for a rigorous analysis of parameter and prediction uncertainties without the need for assuming parameter identifiability or removing non-identifiable parameters. A broad spectrum of MCMC algorithms have been proposed, including single- and multi-chain approaches. However, selecting and tuning sampling algorithms suited for a given problem remains challenging and a comprehensive comparison of different methods is so far not available.ResultsWe present the results of a thorough benchmarking of state-of-the-art single- and multi-chain sampling methods, including Adaptive Metropolis, Delayed Rejection Adaptive Metropolis, Metropolis adjusted Langevin algorithm, Parallel Tempering and Parallel Hierarchical Sampling. Different initialization and adaptation schemes are considered. To ensure a comprehensive and fair comparison, we consider problems with a range of features such as bifurcations, periodical orbits, multistability of steady-state solutions and chaotic regimes. These problem properties give rise to various posterior distributions including uni- and multi-modal distributions and non-normally distributed mode tails. For an objective comparison, we developed a pipeline for the semi-automatic comparison of sampling results.ConclusionThe comparison of MCMC algorithms, initialization and adaptation schemes revealed that overall multi-chain algorithms perform better than single-chain algorithms. In some cases this performance can be further increased by using a preceding multi-start local optimization scheme. These results can inform the selection of sampling methods and the benchmark collection can serve for the evaluation of new algorithms. Furthermore, our results confirm the need to address exploration quality of MCMC chains before applying the commonly used quality measure of effective sample size to prevent false analysis conclusions.Electronic supplementary materialThe online version of this article (doi:10.1186/s12918-017-0433-1) contains supplementary material, which is available to authorized users.

Benchmark Problems for Dynamic Modeling of Intracellular Processes

“…Indeed, the reliability of the findings depends directly on the size and the representativeness of the benchmark collection. Amongst others, previous studies were not able to provide an assessment of the scaling properties Villaverde et al, 2015;Ballnus et al, 2017).…”

“…To ensure that the benchmark problems are realistic and practically relevant, we exclusively included published models and measured experimental data. This is a key difference to existing benchmark collections which mostly considered models with simulated data (Villaverde et al, 2015;Ballnus et al, 2017). The benchmark models possess a broad spectrum of properties (e.g., different types of initial conditions, noise models and inputs), as well as challenges (e.g., structural and practical non-identifiabilities, and objective functions with multiple minima and valleys).…”

Bayesian Data Assimilation to Support Informed Decision Making in Individualized Chemotherapy

“…A popular alternative to SIR in Bayesian inference are MCMC methods with a wide range of different algorithms . MCMC generates an unweighted sample from the posterior by means of a Markov chain (Section 11 in ref.…”

Estimating numbers of fluorescent molecules in single cells by analysing fluctuations in photobleaching

“…To generate samples from the posterior, we use a Metropolis-within-Gibbs scheme [60] with the heterogeneous parameters updated separately from the homogeneous parameters. We employ adaptive parallel tempering to accelerate mixing in the Gibbs sampler [61], which has been found to perform well on benchmark biochemical models [62]. Specifically, we use 10 chains with their temperatures chosen adaptively.…”