Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence

Abstract

Topic models extract groups of words from documents, whose interpretation as a topic hopefully allows for a better understanding of the data. However, the resulting word groups are often not coherent, making them harder to interpret. Recently, neural topic models have shown improvements in overall coherence. Concurrently, contextual embeddings have advanced the state of the art of neural models in general. In this paper, we combine contextualized representations with neural topic models. We find that our approach produces more meaningful and coherent topics than traditional bag-of-words topic models and recent neural models. Our results indicate that future improvements in language models will translate into better topic models.

Local News Online and COVID in the U.S.: Relationships among Coverage, Cases, Deaths, and Audience

“…a variable that can have a non-linear association with the distribution of how prevalent each topic is over time). Notably, we select the structural topic model as opposed to more recent topic modeling frameworks that incorporate more recent innovations in the natural language processing literature, in particular those that incorporate new and more rapid approaches for model optimization (Card, Tan, and Smith 2018) and those that incorporate contextualized word embeddings (Bianchi, Terragni, and Hovy 2021). We opt for the STM over these other approaches for two reasons.…”

Topic Modeling on Podcast Short-Text Metadata

“…The third group has become popular in the last years with the rise of deep learning. Neural topic modeling is based on Variational Auto-Encoders (VAE) [4,17,30,33]. Typically, an encoder such as a MultiLayer Perceptron (MLP) compresses the Bag-of-Words (BoW) document representation into a continuous vector.…”

“…Typically, an encoder such as a MultiLayer Perceptron (MLP) compresses the Bag-of-Words (BoW) document representation into a continuous vector. Then, a decoder reconstructs the document by generating words independently [4,17]. Negative sampling and Quantization Topic Model (NQTM) [33], the latest topic modeling technique on short texts brings two contributions which yielded the current SOTA results.…”

An Empirical Analysis of Topic Models: Uncovering the Relationships between Hyperparameters, Document Length and Performance Measures

“…Following (Bianchi et al, 2021a), we use the contextualized document representations derived from SentenceBERT (Reimers and Gurevych, 2019). We use the pre-trained BERT model fine-tuned on the natural language inference (NLI) task.…”