Robust mixture of experts modeling using thetdistribution

Abstract

Mixture of Experts (MoE) is a popular framework for modeling heterogeneity in data for regression, classification, and clustering. For regression and cluster analyses of continuous data, MoE usually use normal experts following the Gaussian distribution. However, for a set of data containing a group or groups of observations with heavy tails or atypical observations, the use of normal experts is unsuitable and can unduly affect the fit of the MoE model. We introduce a robust MoE modeling using the t distribution. The proposed t MoE (TMoE) deals with these issues regarding heavy-tailed and noisy data. We develop a dedicated expectation-maximization (EM) algorithm to estimate the parameters of the proposed model by monotonically maximizing the observed data log-likelihood. We describe how the presented model can be used in prediction and in model-based clustering of regression data. The proposed model is validated on numerical experiments carried out on simulated data, which show the effectiveness and the robustness of the proposed model in terms of modeling non-linear regression functions as well as in model-based clustering. Then, it is applied to the real-world data of tone perception for musical data analysis, and the one of temperature anomalies for the analysis of climate change data. The obtained results show the usefulness of the TMoE model for practical applications.

A Regularized Mixture of Linear Experts for Quality Prediction in Multimode and Multiphase Industrial Processes

“…However, this criterion can also be employed for model selection. In [36], there is a short discussion on different modeling selections for MoE.…”

Parsimonious mixture‐of‐experts based on mean mixture of multivariate normal distributions

“…Wang et al, 2020), economics (Mazza et al, 2017;Wang et al, 1998) and social sciences (Gormley & Murphy, 2010). There has been a growing interest recently in developing the MoE of univariate regression models based on non-normal distributions as efficient tools for handling non-linear regression problems and model-based clustering of skewed and heavy-tailed distributed regression data; see, for example, Nguyen and McLachlan (2016), Chamroukhi (2016), Chamroukhi (2017) and Mirfarah et al (2021). For the multivariate response, Dang and McNicholas (2015) have proposed MoE of classical MTR models (hereafter called MoE-MTR N ) by modelling the mixing proportions as a function of some covariates.…”

“…Since their beginning more than three decades ago, they have been the subject of numerous research works across various domains such as statistics (Van der Heijden et al, 1996; Wedel, 2002), market segmentation (Dillon & Kutnar, 1994; Gupta & Chintagunta, 1994; Kamakura et al, 1994; Wedel & Kamakura, 2012), computer vision (Theis & Bethge, 2015; X. Wang et al, 2020), economics (Mazza et al, 2017; Wang et al, 1998) and social sciences (Gormley & Murphy, 2010). There has been a growing interest recently in developing the MoE of univariate regression models based on non‐normal distributions as efficient tools for handling non‐linear regression problems and model‐based clustering of skewed and heavy‐tailed distributed regression data; see, for example, Nguyen and McLachlan (2016), Chamroukhi (2016), Chamroukhi (2017) and Mirfarah et al (2021). For the multivariate response, Dang and McNicholas (2015) have proposed MoE of classical MTR models (hereafter called MoE‐MTR) by modelling the mixing proportions as a function of some covariates.…”