Advances in Variational Inference

Zhang, Cheng; Bütepage, Judith; Kjellström, Hedvig; Mandt, Stephan

doi:10.1109/tpami.2018.2889774

Cited by 511 publications

(377 citation statements)

References 93 publications

Supporting

Mentioning

376

Contrasting

Order By: Relevance

“…Graves [21] proposed a stochastic method for variational inference with a diagonal Gaussian posterior that can be applied to almost any differentiable log-loss parametric model, including neural networks. However, there is always a trade-off between complexity of the posterior and scalability and robustness [81]. In this work, we adopt the mean-field variational inference [39].…”

Section: Bayesian Neural Networkmentioning

confidence: 99%

Physics-Informed Probabilistic Learning of Linear Embeddings of Nonlinear Dynamics with Guaranteed Stability

Pan

Duraisamy

2020

SIAM J. Appl. Dyn. Syst.

121

View full text Add to dashboard Cite

The Koopman operator has emerged as a powerful tool for the analysis of nonlinear dynamical systems as it provides coordinate transformations which can globally linearize the dynamics. Recent deep learning approaches such as Linearly-Recurrent Autoencoder Networks (LRAN) show great promise for discovering the Koopman operator for a general nonlinear dynamical system from a data-driven perspective, but several challenges remain. In this work, we formalize the problem of learning the continuous-time Koopman operator with deep neural networks in a measure-theoretic framework. This induces two types of models: differential and recurrent form, the choice of which depends on the availability of the governing equations and data. We then enforce a structural parameterization that renders the realization of the Koopman operator provably stable. A new autoencoder architecture is constructed, such that only the residual of the dynamic mode decomposition is learned. Finally, we employ mean-field variational inference (MFVI) on the aforementioned framework in a hierarchical Bayesian setting to quantify uncertainties in the characterization and prediction of the dynamics of observables. The framework is evaluated on a simple polynomial system, the Duffing oscillator, and an unstable cylinder wake flow with noisy measurements.

show abstract

Section: Bayesian Neural Networkmentioning

confidence: 99%

Physics-Informed Probabilistic Learning of Linear Embeddings of Nonlinear Dynamics with Guaranteed Stability

Pan

Duraisamy

2020

SIAM J. Appl. Dyn. Syst.

121

View full text Add to dashboard Cite

show abstract

“…One can then evaluate the posterior 289 probability of discrete lexical, prosody and speaker states, using the respective likelihood of the (Kim,290 Frisina et al) parameter estimates (and any priors over discrete states should they be available). This MAP 291 scheme can be read in the spirit of predictive coding that has been amortised (Zhang, Butepage et al 2018). 292…”

Section: Model Inversion or Word Recognition 280mentioning

confidence: 99%

Active Listening

Friston

Sajid²,

Quiroga-Martinez³

et al. 2020

Preprint

View full text Add to dashboard Cite

16This paper introduces active listening, as a unified framework for synthesising and recognising speech. The 17 notion of active listening inherits from active inference, which considers perception and action under one 18 universal imperative: to maximise the evidence for our (generative) models of the world. First, we describe 19 a generative model of spoken words that simulates (i) how discrete lexical, prosodic, and speaker attributes 20give rise to continuous acoustic signals; and conversely (ii) how continuous acoustic signals are recognised 21 as words. The 'active' aspect involves (covertly) segmenting spoken sentences and borrows ideas from 22 active vision. It casts speech segmentation as the selection of internal actions, corresponding to the 23 placement of word boundaries. Practically, word boundaries are selected that maximise the evidence for an 24 internal model of how individual words are generated. We establish face validity by simulating speech 25 recognition and showing how the inferred content of a sentence depends on prior beliefs and background 26 noise. Finally, we consider predictive validity by associating neuronal or physiological responses, such as 27 the mismatch negativity and P300, with belief updating under active listening, which is greatest in the 28 absence of accurate prior beliefs about what will be heard next. 29 30 31 Key words: speech recognition, voice, active inference, active listening, segmentation, variational Bayes, 32 audition.'invariant' (Liberman, Cooper et al. 1967)-words are never heard out of a particular context. When 54 considering how words are generated, there is wide variability in the pronunciation of the same word among 55 different speakers (Hillenbrand, Getty et al. 1995, Remez 2010)-and even when spoken by the same 56 speaker, pronunciation depends on prosody (Bänziger and Scherer, 2005). From the perspective of 57 recognition, two signals that are acoustically identical can be perceived as different words or phonemes by 58 human listeners, depending on their context-for example, the preceding words or phonemes (Mann 1980, 59 Miller, Green et al. 1984, preceding spectral content (Holt, Lotto et al. 2000), or the duration of a vowel 60 that follows a consonant (Miller and Liberman 1979). The current approach considers the processes The idea that speech segmentation and lexical inference operate together did not figure in early accounts of 63 speech recognition. For example, the Fuzzy Logic Model of Perception (FLMP) (Oden and Massaro 1978, 64 Massaro 1987, Massaro 1989 matches acoustic features with prototype representations to recognise 65 phonemes, even when considered in the context of words and sentences. Similarly, the Neighbourhood 66 Activation Model (NAM) (Luce 1986, Luce and Pisoni 1998) considers individual word recognition; it 67 accounts for effects of word frequency, but does not address the segmentation problem. Later connectionist 68 accounts, such as TRACE (McClelland and Elman 1986), assumed that competition between lexical nodes ...

show abstract

“…The above models were implemented using the Python-based probabilistic programming language PyMC3 v3.8 [36] and inference was conducted using Automatic Differentiation Vari-ational Inference (ADVI; [37]), instead of developing bespoke estimation algorithms, which is a rather laborious process particularly when multiple candidate models are considered [38][39][40]. Variational inference (VI; [41,42]) is a computationally efficient approach for Bayesian inference, which aims to approximate the posterior density p(z|y) of latent variables z given data y using a surrogate probability density q θ (z) parametrised by a vector of variational parameters θ. In our case, the data y are the locus-and sample-specific read counts r ij and R ij , the local copy numbers D ij , the sample-specific purities ρ j and the sample collection times t j , while the latent variables z are the cancer cell fractions φ jk , the cluster weights w k , the amplitudes h 2 k , the time-scales τ k and the sample-specific dispersions v j .…”

Section: Inferencementioning

confidence: 99%

A statistical approach for tracking clonal dynamics in cancer using longitudinal next-generation sequencing data

Vavoulis

Cutts

Taylor

et al. 2020

Preprint

View full text Add to dashboard Cite

Tumours are composed of genotypically and phenotypically distinct cancer cell populations (clones), which are subject to a process of Darwinian evolution in response to changes in their local microenvironment, such as drug treatment. In a cancer patient, this process of continuous adaptation can be studied through next-generation sequencing of multiple tumour samples combined with appropriate bioinformatics and statistical methodologies. One family of statistical methods for clonal deconvolution seeks to identify groups of mutations and estimate the prevalence of each group in the tumour, while taking into account its purity and copy number profile. These methods have been used in the analysis of cross-sectional data, as well as for longitudinal data by discarding information on the timing of sample collection. Two key questions are how (in the case of longitudinal data) can we incorporate such information in our analyses and if there is any benefit in doing so. Regarding the first question, we incorporated information on the temporal spacing of longitudinally collected samples into standard non-parametric approaches for clonal deconvolution by modelling the time dependence of the prevalence of each clone as a Gaussian process. This permitted reconstruction of the temporal profile of the abundance of each clone continuously from several sparsely collected samples and without any strong prior assumptions on the functional form of this profile. Regarding the second question, we tested various model configurations on a range of whole genome, whole exome and targeted sequencing data from patients with chronic lymphocytic leukaemia, on liquid biopsy data from a patient with melanoma and on synthetic data. We demonstrate that incorporating temporal information in our analysis improves model performance, as long as data of sufficient volume and complexity are available for estimating free model parameters. We expect that our approach will be useful in cases where collecting a relatively long sequence of tumour samples is feasible, as in the case of liquid cancers (e.g. leukaemia) and liquid biopsies. The statistical methodology presented in this paper is freely available at github.com/dvav/clonosGP.

show abstract

Advances in Variational Inference

Cited by 511 publications

References 93 publications

Physics-Informed Probabilistic Learning of Linear Embeddings of Nonlinear Dynamics with Guaranteed Stability

Physics-Informed Probabilistic Learning of Linear Embeddings of Nonlinear Dynamics with Guaranteed Stability

Active Listening

A statistical approach for tracking clonal dynamics in cancer using longitudinal next-generation sequencing data

Contact Info

Product

Resources

About