Acceleration of the EM Algorithm by using Quasi-Newton Methods

Existing data mining techniques, more particularly iterative learning algorithms, become overwhelmed with big data. While parallelism is an obvious and, usually, necessary strategy, we observe that both (1) continually revisiting data and (2) visiting all data are two of the most prominent problems especially for iterative, unsupervised algorithms like expectation maximization algorithm for clustering (EM-T). Our strategy is to embed EM-T into a nonlinear hierarchical data structure (heap) that allows us to (1) separate data that needs to be revisited from data that does not and (2) narrow the iteration toward the data that is more difficult to cluster. We call this extended EM-T, EM*. We show our EM* algorithm outperform EM-T algorithm over large real-world and synthetic data sets. We lastly conclude with some theoretical underpinnings that explain why EM* is successful.

show abstract

“…[23] uses conjugate gradient acceleration and [24] uses QN. Using line search, unlike the pure methods, these are globally convergent.…”

Section: Related and Previous Workmentioning

confidence: 99%

Using data to build a better EM: EM* for big data

Kurban

Jenne

Dalkılıç

2017

Int J Data Sci Anal

View full text Add to dashboard Cite

show abstract

“…Other modifications involve approximations to derivatives 7 of the likelihood to yield Quasi-Newton e.g. [13,22] or gradient type procedures e.g. [12,18].…”

Section: Expectation Maximisation Algorithmmentioning

confidence: 99%

Parameter expansion for estimation of reduced rank covariance matrices(Open Access publication)

Meyer

2007

Genet. Sel. Evol.

View full text Add to dashboard Cite

-Parameter expanded and standard expectation maximisation algorithms are described for reduced rank estimation of covariance matrices by restricted maximum likelihood, fitting the leading principal components only. Convergence behaviour of these algorithms is examined for several examples and contrasted to that of the average information algorithm, and implications for practical analyses are discussed. It is shown that expectation maximisation type algorithms are readily adapted to reduced rank estimation and converge reliably. However, as is well known for the full rank case, the convergence is linear and thus slow. Hence, these algorithms are most useful in combination with the quadratically convergent average information algorithm, in particular in the initial stages of an iterative solution scheme.restricted maximum likelihood / reduced rank estimation / algorithms / expectation maximisation / average information

show abstract

“…When both the gradient and Hessian are calculated analytically, the algorithm is referred to as NewtonRaphson. A less complicated but reliable approach is to compute the derivatives for the gradient vector analytically and approximate the Hessian matrix by numerical differentiation of the gradient vector (Dennis & Schnabel, 1986;Jamshidian & Jennrich, 1997); in this case, the procedure is termed quasi-Newton. The latter approach is considered here.…”

Section: Maximum-likelihood Estimation Of Model Parametersmentioning

confidence: 99%

“…may be approximated by numerical differentiation of g( ) (Dennis & Schnabel, 1986;Jamshidian & Jennrich, 1997). This eases the technical burden of specifying the matrix.…”

Section: Author Notementioning

confidence: 99%

A Newton procedure for conditionally linear mixed-effects models

Blozis

2007

Behavior Research Methods

View full text Add to dashboard Cite

Mixed-effects models, also called multilevel models and hierarchical models, provide a framework for the study of clustered data when randomly selected units of study are nested within higher-order units that are also randomly selected. Hierarchical data structures occur in many domains, including psychology, sociology, business, nursing, and medicine. A classic example is education, where students are "nested" within classrooms. This general methodology addresses the dependencies of scores within random clusters by allowing coefficients defining a data model to vary across clusters. Many books are devoted to the study of these models, including Davidian and Giltinan (1995), Hox (2002), Kreft and de Leeuw (1998), Longford (1993), Raudenbush and Bryk (2002), Singer and Willett (2003), and Snijders and Bosker (1999). A mixed-effects model allows for both the study of variables at multiple levels and for interactions between variables at different levels. Estimation of effects at the different levels is simultaneous. The methodology is flexible, allowing for unbalanced data, such as unequal group sizes.Given the adaptable nature of this approach, mixedeffects models are increasingly used in areas where stratified sampling is common. This is also due in part to the widespread availability of statistical software packages for estimating these models. These include HLM Hedeker's programs for a variety of mixed-effects models (available for downloading at tigger.uic.edu/~hedeker/ mix.html).These models are also applied to repeated measures and longitudinal data in which repeated measures of the same variable are taken, for example, within a single experimental session or across some specified time period. Such data are hierarchically structured because the repeated measures are nested within individuals. Because of the flexibility of the methodology, individuals need not be assessed the same number of times or measured (in the context of time-structured data) according to the same measurement occasions (Jennrich & Schluchter, 1986). Thus, unbalanced data and unequal spacing between assessments (for time-structured data) are easily handled. This means the methodology has greater utility than classic methods, such as a repeated measures ANOVA, which requires complete, time-balanced data. In cases of timestructured data in particular, taking into account accurate measures of time may be important in many studies in which change is a primary focus (e.g., Collins & Graham, 2002).In the context of longitudinal data in particular, the models are often called latent curve models (Meredith & Tisak, 1984, 1990. In a latent curve model, an individual's response is typically studied as a function of time with the addition of time-specific error. Coefficients characterizing the response over time, such as an intercept and linear time effect when change is linear, may be individual-specific; for instance, each individual may have a unique intercept and time effect. These individual-specific effects, called random coefficients or r...

show abstract

Acceleration of the EM Algorithm by using Quasi-Newton Methods

Cited by 184 publications

References 27 publications

Using data to build a better EM: EM* for big data

Using data to build a better EM: EM* for big data

Parameter expansion for estimation of reduced rank covariance matrices(Open Access publication)

A Newton procedure for conditionally linear mixed-effects models

Contact Info

Product

Resources

About