Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networks

Marvin, Schmitt,; Bürkner, Paul‐Christian; Köthe, Ullrich; Radev, Stefan T.

doi:10.48550/arxiv.2112.08866

Cited by 9 publications

(17 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2d, see Supplementary material for details). Recent methodological work in SBI addresses this problem, e.g., by automatically detecting model misspecification ( Schmitt et al, 2022 ) or by explicitly incorporating the model mismatch into the generative model ( Ward et al, 2022 ).…”

Section: Discussionmentioning

confidence: 99%

Simulation-based inference for efficient identification of generative models in computational connectomics

Boelts

Harth

Gao

et al. 2023

Preprint

View full text Add to dashboard Cite

Recent advances in connectomics research enable the acquisition of increasing amounts of data about the connectivity patterns of neurons. How can we use this wealth of data to efficiently derive and test hypotheses about the principles underlying these patterns? A common approach is to simulate neural networks using a hypothesized wiring rule in a generative model and to compare the resulting synthetic data with empirical data. However, most wiring rules have at least some free parameters and identifying parameters that reproduce empirical data can be challenging as it often requires manual parameter tuning. Here, we propose to use simulation-based Bayesian inference (SBI) to address this challenge. Rather than optimizing a single rule to fit the empirical data, SBI considers many parametrizations of a wiring rule and performs Bayesian inference to identify the parameters that are compatible with the data. It uses simulated data from multiple candidate wiring rules and relies on machine learning methods to estimate a probability distribution (the `posterior distribution over rule parameters conditioned on the data') that characterizes all data-compatible rules. We demonstrate how to apply SBI in connectomics by inferring the parameters of wiring rules in anin silicomodel of the rat barrel cortex, givenin vivoconnectivity measurements. SBI identifies a wide range of wiring rule parameters that reproduce the measurements. We show how access to the posterior distribution over all data-compatible parameters allows us to analyze their relationship, revealing biologically plausible parameter interactions and enabling experimentally testable predictions. We further show how SBI can be applied to wiring rules at different spatial scales to quantitatively rule out invalid wiring hypotheses. Our approach is applicable to a wide range of generative models used in connectomics, providing a quantitative and efficient way to constrain model parameters with empirical connectivity data.

show abstract

Section: Discussionmentioning

confidence: 99%

Simulation-based inference for efficient identification of generative models in computational connectomics

Boelts

Harth

Gao

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Recent research has found neural SBI methods behave poorly under model misspecification (Bon et al, 2022;Cannon et al, 2022;Schmitt et al, 2021;Ward et al, 2022). It is not surprising that neural SBI methods suffer from the same issues as ABC and BSL when compatibility is not satisfied as they are based on many of the same principles.…”

Section: Robust Extension To Sequential Neural Likelihoodmentioning

confidence: 99%

“…Recently, methods have been developed to detect model misspecification when applying neural posterior estimation for both amortised (Ward et al, 2022) and sequential (Schmitt et al, 2021) approaches. 1 Schmitt et al (2021) use a maximum mean discrepancy (MMD) estimator to detect a "simulation gap" between the observed and simulated data.…”

Section: Robust Extension To Sequential Neural Likelihoodmentioning

confidence: 99%

“…However, the machine learning approaches, such as sequential neural likelihood (SNL) and sequential neural posterior (SNP) have been shown to exhibit poor performance under model misspecification (e.g. Bon et al (2022); Cannon et al (2022); Schmitt et al (2021); Ward et al (2022)). Thus there is a critical need to develop these neural approaches so they are robust to model misspecification.…”

Section: Introductionmentioning

confidence: 99%

“…Schmitt et al (2021) also add robustness model misspecification to BayesFlow(Radev et al, 2022). BayesFlow, like NPE, is an amortised neural approximation of the posterior.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Misspecification-robust Sequential Neural Likelihood

Kelly¹,

Nott²,

Frazier³

et al. 2023

Preprint

View full text Add to dashboard Cite

Simulation-based inference (SBI) techniques are now an essential tool for the parameter estimation of mechanistic and simulatable models with intractable likelihoods. Statistical approaches to SBI such as approximate Bayesian computation and Bayesian synthetic likelihood have been well studied in the well specified and misspecified settings. However, most implementations are inefficient in that many model simulations are wasted. Neural approaches such as sequential neural likelihood (SNL) have been developed that exploit all model simulations to build a surrogate of the likelihood function. However, SNL approaches have been shown to perform poorly under model misspecification. In this paper, we develop a new method for SNL that is robust to model misspecification and can identify areas where the model is deficient. We demonstrate the usefulness of the new approach on several illustrative examples.

show abstract

A General Integrative Neurocognitive Modeling Framework to Jointly Describe EEG and Decision-making on Single Trials

2023

View full text Add to dashboard Cite

Despite advances in techniques for exploring reciprocity in brain-behavior relations, few studies focus on building neurocognitive models that describe both human EEG and behavioral modalities at the single-trial level. Here, we introduce a new integrative joint modeling framework for the simultaneous description of single-trial EEG measures and cognitive modeling parameters of decision-making. As specific examples, we formalized how single-trial N200 latencies and centro-parietal positivities (CPPs) are predicted by changing single-trial parameters of various drift-diffusion models (DDMs). We trained deep neural networks to learn Bayesian posterior distributions of unobserved neurocognitive parameters based on model simulations. These models do not have closed-form likelihoods and are not easy to fit using Markov chain Monte Carlo (MCMC) methods because nuisance parameters on single trials are shared in both behavior and neural activity. We then used parameter recovery assessment and model misspecification to ascertain how robustly the models’ parameters can be estimated. Moreover, we fit the models to three different real datasets to test their applicability. Finally, we provide some evidence that single-trial integrative joint models are superior to traditional integrative models. The current single-trial paradigm and the simulation-based (likelihood-free) approach for parameter recovery can inspire scientists and modelers to conveniently develop new neurocognitive models for other neural measures and to evaluate them appropriately.

show abstract

Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networks

Cited by 9 publications

References 15 publications

Simulation-based inference for efficient identification of generative models in computational connectomics

Simulation-based inference for efficient identification of generative models in computational connectomics

Misspecification-robust Sequential Neural Likelihood

A General Integrative Neurocognitive Modeling Framework to Jointly Describe EEG and Decision-making on Single Trials

Contact Info

Product

Resources

About