Penalized Regression Splines in Mixture Density Networks

Abstract

Mixture Density Networks (MDN) belong to a class of models that can be applied to data which cannot be sufficiently described by a single distribution since it originates from different components of the main unit and therefore needs to be described by a mixture of densities. In some situations, however, MDNs seem to have problems with the proper identification of the latent components. While these identification issues can to some extent be contained by using custom initialization strategies for the network weights, this solution is still less than ideal since it involves subjective opinions. We therefore suggest replacing the hidden layers between the model input and the output parameter vector of MDNs and estimating the respective distributional parameters with penalized cubic regression splines. Applying this approach to data from Gaussian mixture distributions as well gamma mixture distributions proved to be successful with the identification issues not playing a role anymore and the splines reliably converging to the true parameter values.

Confidence Intervals for Finite Mixture Regression Based on Resampling Techniques

“…For datasets with a large number of observations, investigating the suitability of our approach for a recently proposed EM-algorithm trained on mini-batches (Nguyen et al, 2020) is an important next step to assess its practicability in the context of large-scale datasets. This also encompasses the adaptation to the framework of mixture density networks with interpretable effect estimates as for example implemented in Hepp et al (2022) and Seifert et al (2022). In contrast to Expectation-Maximization, these methods estimate the posterior class probabilities based on the gradient of the loss-function, but the general concept of the proposed resampling strategy should still be applicable.…”

Estimating and evaluating mixed and semiparametric models with statistical and deep learning methods

“…Another possible extension would be the adaptation to mixture density networks, as e.g. done by Seifert et al (2022). Another possible focus is to switch our approach to a Bayesian-based training approach.…”

“…Chang et al (2021) introduced NODE-GAM, a differentiable model based on forgetful decision trees developed for high-risk domains. All these models follow the additive framework from GAMs and learn the nonlinear additive features with separate networks, one for each feature or feature interaction, either leveraging MLPs (Potts, 1999;de Waal and du Toit, 2007;Agarwal et al, 2021;Yang et al, 2021;Radenovic et al, 2022), using decision trees (Chang et al, 2021) or using Splines (Rügamer et al, 2020;Seifert et al, 2022;Luber et al, 2023).…”