New estimation and feature selection methods in mixture‐of‐experts models

Khalili, Abbas

doi:10.1002/cjs.10083

Cited by 40 publications

(44 citation statements)

References 31 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is well documented that maximum likelihood (ML) estimation can be used to obtain consistent estimates of the mean function, when the number of components g is known; see [18] for results regarding the ML estimator for MoE models with normal component PDFs, [9] for results regarding the ML estimator when the component PDFs are from the oneparameter exponential family, and [14] for the case of Laplace component PDFs. Results regarding regularized ML estimation of MoE models were obtained in [11]. In [14] and [5], the Bayesian information criterion [15] is demonstrated to be effective for the determination of an unknown g.…”

Section: Discussionmentioning

confidence: 99%

A Universal Approximation Theorem for Mixture-of-Experts Models

2016

View full text Add to dashboard Cite

The mixture of experts (MoE) model is a popular neural network architecture for nonlinear regression and classification. The class of MoE mean functions is known to be uniformly convergent to any unknown target function, assuming that the target function is from Sobolev space that is sufficiently differentiable and that the domain of estimation is a compact unit hypercube. We provide an alternative result, which shows that the class of MoE mean functions is dense in the class of all continuous functions over arbitrary compact domains of estimation. Our result can be viewed as a universal approximation theorem for MoE models.

show abstract

Section: Discussionmentioning

confidence: 99%

A Universal Approximation Theorem for Mixture-of-Experts Models

2016

View full text Add to dashboard Cite

show abstract

“…Furthermore, the added estimation of the covariate-dependent class probabilities introduces excess variability that is likely to adversely effect the performance of these models under the assumptions of the problem. Similarly to the regular FMR, a penalized HME using generalized linear models as experts (Khalili, 2010) requires the definition of the correct structure for the linear predictor.…”

Section: Introductionmentioning

confidence: 99%

Prediction with a flexible finite mixture-of-regressions

Ahonen

Nevalainen

Larocque

2019

Computational Statistics & Data Analysis

View full text Add to dashboard Cite

Finite mixture regression (FMR) is widely used for modeling data that originate from heterogeneous populations. In these settings, FMR can offer increased predictive power compared to more traditional one-class models. However, existing FMR methods rely heavily on mixtures of linear models, where the linear predictor must be given as an input. A flexible FMR model is presented using a combination of the random forest learner and a penalized linear FMR. The performance of the new method is assessed by predictive log-likelihood in extensive simulation studies. The method is shown to achieve equal performance with the existing FMR methods when the true regression functions are in fact linear and superior performance in cases where at least one of the regression functions is nonlinear. The method can handle a large number of covariates, and its predictive ability is not greatly affected by surplus variables.

show abstract

“…The L-MLR was then generalized to the divergent number of variables setting in Khalili and Lin (2013), and to the mixture of experts setting in Khalili (2010).…”

Section: Introductionmentioning

confidence: 99%

A globally convergent algorithm for lasso-penalized mixture of linear regression models

Lloyd‐Jones

Nguyen

McLachlan

2018

Computational Statistics & Data Analysis

View full text Add to dashboard Cite

Variable selection is an old and pervasive problem in regression analysis. One solution is to impose a lasso penalty to shrink parameter estimates toward zero and perform continuous model selection. The lasso-penalized mixture of linear regressions model (L-MLR) is a class of regularization methods for the model selection problem in the fixed number of variables setting. In this article, we propose a new algorithm for the maximum penalized-likelihood estimation of the L-MLR model. This algorithm is constructed via the minorization-maximization algorithm paradigm. Such a construction allows for coordinate-wise updates of the parameter components, and produces globally convergent sequences of estimates that generate monotonic sequences of penalized log-likelihood values. These three features are missing in the previously presented approximate expectation-maximization algorithms. The previous difficulty in producing a globally convergent algorithm for the maximum penalized-likelihood estimation of the L-MLR model is due to the intractability of finding exact updates for the mixture model mixing proportions in the maximization-step. In our algorithm, we solve this issue by showing that it can be converted into a polynomial root finding problem. Our solution to this problem involves a polynomial basis conversion that is interesting in its own right. The method is tested in simulation and with an application to Major League Baseball salary data from the 1990s and the present day. We explore the concept of whether player salaries are associated with batting performance.

show abstract

New estimation and feature selection methods in mixture‐of‐experts models

Cited by 40 publications

References 31 publications

A Universal Approximation Theorem for Mixture-of-Experts Models

A Universal Approximation Theorem for Mixture-of-Experts Models

Prediction with a flexible finite mixture-of-regressions

A globally convergent algorithm for lasso-penalized mixture of linear regression models

Contact Info

Product

Resources

About