An introduction to<scp>Majorization‐Minimization</scp>algorithms for machine learning and statistical estimation

Nguyen, Hien D.

doi:10.1002/widm.1198

Cited by 17 publications

(9 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As such, iterative or numerical schemes are often employed to conduct maximization. In Nguyen and McLachlan () and Nguyen and McLachlan (), the authors considered the blockwise‐MM (minorization–maximization) algorithm framework of Lange (); see Nguyen, for a concise tutorial on MM algorithms.…”

Section: Theoretical Resultsmentioning

confidence: 99%

Practical and theoretical aspects of mixture‐of‐experts modeling: An overview

Nguyen

Chamroukhi

2018

WIREs Data Min & Knowl

Self Cite

View full text Add to dashboard Cite

Mixture-of-experts (MoE) models are a powerful paradigm for modeling data arising from complex data generating processes (DGPs). In this article, we demonstrate how different MoE models can be constructed to approximate the underlying DGPs of arbitrary types of data. Due to the probabilistic nature of MoE models, we propose the maximum quasi-likelihood (MQL) approach as a method for estimating MoE model parameters from data, and we provide conditions under which MQL estimators are consistent and asymptotically normal. The blockwise minorizationmaximization (blockwise-MM) algorithm framework is proposed as an all-purpose method for constructing algorithms for obtaining MQL estimators. An example derivation of a blockwise-MM algorithm is provided. We then present a method for constructing information criteria for estimating the number of components in MoE models and provide justification for the classic Bayesian information criterion (BIC). We explain how MoE models can be used to conduct classification, clustering, and regression and illustrate these applications via two worked examples.

show abstract

Section: Theoretical Resultsmentioning

confidence: 99%

Practical and theoretical aspects of mixture‐of‐experts modeling: An overview

Nguyen

Chamroukhi

2018

WIREs Data Min & Knowl

Self Cite

View full text Add to dashboard Cite

show abstract

“…Using the framework of block-successive lower bound maximization algorithms (also known as block minormizationmaximization algorithms; cf. Hunter andLange, 2004 andNguyen, 2017), Nguyen and Wood [2016b] proposed the following iterative algorithm for the computation of (3), upon observing a realization x 1 , . .…”

Section: The Block-successive Lower Bound Maximization Algorithmmentioning

confidence: 99%

The Fully-Visible Boltzmann Machine and the Senate of the 45th Australian Parliament in 2016

et al. 2018

View full text Add to dashboard Cite

After the 2016 double dissolution election, the 45th Australian Parliament was formed. At the time of its swearing in, the Senate of the 45th Australian Parliament consisted of nine political parties, the largest number in the history of the Australian Parliament. Due to the breadth of the political spectrum that the Senate represented, the situation presented an interesting opportunity for the study of political interactions in the Australian context. Using publicly available Senate voting data in 2016, we quantitatively analyzed two aspects of the Senate. Firstly, we analyzed the degree to which each of the non-government parties of the Senate are pro-or anti-government. Secondly, we analyzed the degree to which the votes of each of the non-government Senate parties are in concordance or discordance with one another. We utilized the fully-visible Boltzmann machine (FVBM) model in order to conduct these analyses. The FVBM is an artificial neural network that can be viewed as a multivariate generalization of the Bernoulli distribution. Via a maximum pseudolikelihood estimation approach, we conducted parameter estimation and constructed hypotheses test that revealed the interaction structures within the Australian Senate. The conclusions that we drew are well-supported by external sources of information.

show abstract

“…• and machine learning: nonnegative matrix factorization (Lee and Seung, 1999), matrix completion (Mazumder et al, 2010;Chi et al, 2013), clustering Xu and Lange, 2019), discriminant analysis (Wu and Lange, 2010), support vector machines (Nguyen, 2017a).…”

Section: Introductionmentioning

confidence: 99%

Nonconvex Optimization via MM Algorithms: Convergence Theory

Lange

Won

Landeros

et al. 2021

Wiley StatsRef: Statistics Reference Online

View full text Add to dashboard Cite

The majorization-minimization (MM) principle is an extremely general framework for deriving optimization algorithms. It includes the expectation-maximization (EM) algorithm, proximal gradient algorithm, concave-convex procedure, quadratic lower bound algorithm, and proximal distance algorithm as special cases. Besides numerous applications in statistics, optimization, and imaging, the MM principle finds wide applications in large scale machine learning problems such as matrix completion, discriminant analysis, and nonnegative matrix factorizations. When applied to nonconvex optimization problems, MM algorithms enjoy the advantages of convexifying the objective function, separating variables, numerical stability, and ease of implementation. However, compared to the large body of literature on other optimization algorithms, the convergence analysis of MM algorithms is scattered and problem specific. This survey presents a unified treatment of the convergence of MM algorithms. With modern applications in mind, the results encompass non-smooth objective functions and nonasymptotic analysis.

show abstract

An introduction toMajorization‐Minimizationalgorithms for machine learning and statistical estimation

Cited by 17 publications

References 59 publications

Practical and theoretical aspects of mixture‐of‐experts modeling: An overview

Practical and theoretical aspects of mixture‐of‐experts modeling: An overview

The Fully-Visible Boltzmann Machine and the Senate of the 45th Australian Parliament in 2016

Nonconvex Optimization via MM Algorithms: Convergence Theory

Contact Info

Product

Resources

About