Convergence of the Expectation-Maximization Algorithm Through Discrete-Time Lyapunov Stability Theory

Romero, Orlando; Chatterjee, Sarthak; Pequito, Sérgio

doi:10.23919/acc.2019.8814665

Cited by 9 publications

(7 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our work aims further to highlight that data augmentation frameworks in settings where the amount of augmented data dominates the number of observations may lead to more accurate inferences by incorporating domain knowledge or other type of information in the augmentation (like here information regarding the geometry of the system's invariant density). Many of the algorithms employed with data augmentation frameworks exhibit only local convergence, e.g., the Expectation Maximisation algorithm employed here [81]. In settings where the initial estimate strongly deviates from its true value, naive data augmentation strategies might therefore converge to sub-optimal solutions, that do not reflect the ground truth.…”

Section: Broader Impact Statementmentioning

confidence: 99%

Geometric path augmentation for inference of sparsely observed stochastic nonlinear systems

Maoutsa¹

2023

Preprint

View full text Add to dashboard Cite

Stochastic evolution equations describing the dynamics of systems under the influence of both deterministic and stochastic forces are prevalent in all fields of science. Yet, identifying these systems from sparse-in-time observations remains still a challenging endeavour. Existing approaches focus either on the temporal structure of the observations by relying on conditional expectations, discarding thereby information ingrained in the geometry of the system's invariant density; or employ geometric approximations of the invariant density, which are nevertheless restricted to systems with conservative forces. Here we propose a method that reconciles these two paradigms. We introduce a new data-driven path augmentation scheme that takes the local observation geometry into account. By employing non-parametric inference on the augmented paths, we can efficiently identify the deterministic driving forces of the underlying system for systems observed at low sampling rates. * https://dimitra-maoutsa.gitlab.io/ 2 Here we employ the term path augmentation to refer to what is widely known as data augmentation. We resorted to this term because we consider it as more elegant and better descriptive of the proposed augmentation process, while the term 'data augmentation' is considerably vague.

show abstract

Section: Broader Impact Statementmentioning

confidence: 99%

Geometric path augmentation for inference of sparsely observed stochastic nonlinear systems

Maoutsa¹

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…If x * is a fixed point of this system, using Taylor's theorem we have for some T 0 > 0, (x t+1 − x * ) ≈ g(x * )(x t − x * ) for all t ≥ T 0 . Recall that the linear rate of convergence [Romero et al, 2019] is given by, γ = lim t→∞ x t+1 − x * / x t − x * , provided the limit exists. In the above scenario, the iterates converge when g(x * ) < 1 and the rate of convergence is g(x * ).…”

Section: Asymptotic Stability Of Tangent Transform Emmentioning

confidence: 99%

Statistical optimality and stability of tangent transform algorithms in logit models

Ghosh,

Bhattacharya,

Pati

2020

Preprint

View full text Add to dashboard Cite

A systematic approach to finding variational approximation in an otherwise intractable nonconjugate model is to exploit the general principle of convex duality by minorizing the marginal likelihood that renders the problem tractable. While such approaches are popular in the context of variational inference in non-conjugate Bayesian models, theoretical guarantees on statistical optimality and algorithmic convergence are lacking. Focusing on logistic regression models, we provide mild conditions on the data generating process to derive non-asymptotic upper bounds to the risk incurred by the variational optima. We demonstrate that these assumptions can be completely relaxed if one considers a slight variation of the algorithm by raising the likelihood to a fractional power. Next, we utilize the theory of dynamical systems to provide convergence guarantees for such algorithms in logistic and multinomial logit regression. In particular, we establish local asymptotic stability of the algorithm without any assumptions on the datagenerating process. We explore a special case involving a semi-orthogonal design under which a global convergence is obtained. The theory is further illustrated using several numerical studies.

show abstract

“…While a significant volume of these optimization-based papers have been published at machine learning (ML) venues, only a few have been explicitly dedicated to addressing concrete ML problems, applications, or algorithms [12,13,14,15,16]. Furthermore, only a small subset of this emerging topic of research has focused directly on discrete-time analysis that is the direct result of discretizations of an underlying continuous-time version of the algorithms [3,17,18,8,19].…”

Section: Introductionmentioning

confidence: 99%

A Dynamical Systems Approach for Convergence of the Bayesian EM Algorithm

Romero,

Das,

Chen

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Out of the recent advances in systems and control (S&C)-based analysis of optimization algorithms, not enough work has been specifically dedicated to machine learning (ML) algorithms and its applications. This paper addresses this gap by illustrating how (discrete-time) Lyapunov stability theory can serve as a powerful tool to aid, or even lead, in the analysis (and potential design) of optimization algorithms that are not necessarily gradient-based. The particular ML problem that this paper focuses on is that of parameter estimation in an incomplete-data Bayesian framework via the popular optimization algorithm known as maximum a posteriori expectation-maximization (MAP-EM). Following first principles from dynamical systems stability theory, conditions for convergence of MAP-EM are developed. Furthermore, if additional assumptions are met, we show that fast convergence (linear or quadratic) is achieved, which could have been difficult to unveil without our adopted S&C approach. The convergence guarantees in this paper effectively expand the set of sufficient conditions for EM applications, thereby demonstrating the potential of similar S&C-based convergence analysis of other ML algorithms.Furthermore, most literature in S&C regarding Lyapunov stability theory deals exclusively with continuous-time systems, and (dynamical) stability properties are typically not preserved after discretization [22]. However, a vast amount of general Lyapunov-based results possess a discrete-time equivalent [

show abstract

Convergence of the Expectation-Maximization Algorithm Through Discrete-Time Lyapunov Stability Theory

Cited by 9 publications

References 11 publications

Geometric path augmentation for inference of sparsely observed stochastic nonlinear systems

Geometric path augmentation for inference of sparsely observed stochastic nonlinear systems

Statistical optimality and stability of tangent transform algorithms in logit models

A Dynamical Systems Approach for Convergence of the Bayesian EM Algorithm

Contact Info

Product

Resources

About