On Learning Prediction-Focused Mixtures

Abstract

Probabilistic models help us encode latent structures that both model the data and are ideally also useful for specific downstream tasks. Among these, mixture models and their time-series counterparts, hidden Markov models, identify discrete components in the data. In this work, we focus on a constrained capacity setting, where we want to learn a model with relatively few components (e.g. for interpretability purposes). To maintain prediction performance, we introduce prediction-focused modeling for mixtures, which automatically selects the dimensions relevant to the prediction task. Our approach identifies relevant signal from the input, outperforms models that are not prediction-focused, and is easy to optimize; we also characterize when prediction-focused modeling can be expected to work.

A Joint Learning Approach for Semi-supervised Neural Topic Modeling

“…The upstream setting allows us to implicitly train our classifier and topic model in a one-stage setting that is end-to-end. This has the benefit of allowing us to tune the trade-off between our classifier and topic model performance in a predictionconstrained framework, which has been shown to achieve better empirical results when latent variable models are used as a dimensionality reduction tool (Hughes et al, 2018;Sharma et al, 2021). Furthermore, the upstream setting allows us to introduce the document label classifier as a latent variable, enabling our model to work in semisupervised settings.…”