Collaboratively Modeling and Embedding of Latent Topics for Short Texts

Abstract

Deriving a successful document representation is the critical challenge in many downstream tasks in NLP, especially when documents are very short. It is challenging to handle the sparsity and the noise problems confronting short texts. Some approaches employ latent topic models, based on global word co-occurrence, to obtain topic distribution as the representation. Others leverage word embeddings, which consider local conditional dependencies, to map a document as a summation vector of them. Unlike the existing works which explore the strategy of utilizing one to help the other, i.e., topic models for word embeddings or vice versa, we propose CME-DMM, a collaboratively modeling and embedding framework for capturing coherent latent topics from short texts. CME-DMM incorporates topic and word embeddings through the attention mechanism and implants them into the latent topic models, which significantly improve the quality of latent topics. Extensive experiments demonstrate that CME-DMM could perceive more coherent topics than other popular methods, resulting in a better performance in downstream NLP tasks such as classification. Besides the interpretable latent topics, the corresponding topic embeddings can describe the meanings of latent topics in the semantic space. The attention vectors, as a by-product of the learning process, can identify the keywords in noisy short texts.

Enhancing Big Social Media Data Quality for Use in Short-Text Topic Modeling

“…These models seem to outperform both the DMM and individual PDMM methods but also involve high computation costs. A new model called a Collaboratively Modeling and Embedding DMM (CME-DMM) was proposed by Liu [67] for capturing coherent hidden topics from STs. All these models were suggested for topic modeling over short text.…”

A Novel Framework to Detect Irrelevant Software Requirements Based on MultiPhiLDA as the Topic Model

“…Liu et al [22] proposed Collaboratively Modeling and Embedding -Dirichlet Multinomial Mixtures (CME-DMM) to apply topic modeling on short texts that were collected from a social media platform. Liu identified major challenges in modeling latent topics in short texts, including the inadequacy of word co-occurrence instances, the need to capture local contextual dependencies in noisy short texts due to their limited length, and maintaining consistency between the latent topic models based on word co-occcurrences and their representation based on local contextual dependencies.…”

Behavior Analysis of Pandemic Source Media Communications

“…Topic modeling is well known for text mining researchers in Natural Language Processing (NLP), which aims to extract and interpret knowledge from data [13]. A popular method, such as Latent Dirichlet Allocation (LDA) has been proven beneficial in finding useful words that are associated with topics [13].…”