Topic Modeling over Short Texts by Incorporating Word Embeddings

Abstract

Inferring topics from the overwhelming amount of short texts becomes a critical but challenging task for many content analysis tasks, such as content charactering, user interest profiling, and emerging topic detecting. Existing methods such as probabilistic latent semantic analysis (PLSA) and latent Dirichlet allocation (LDA) cannot solve this problem very well since only very limited word co-occurrence information is available in short texts. This paper studies how to incorporate the external word correlation knowledge into short texts to improve the coherence of topic modeling. Based on recent results in word embeddings that learn semantically representations for words from a large corpus, we introduce a novel method, Embedding-based Topic Model (ETM), to learn latent topics from short texts. ETM not only solves the problem of very limited word co-occurrence information by aggregating short texts into long pseudotexts, but also utilizes a Markov Random Field regularized model that gives correlated words a better chance to be put into the same topic. The experiments on real-world datasets validate the effectiveness of our model comparing with the state-of-the-art models.

A Detailed Survey on Topic Modeling for Document and Short Text Data

“…Data are collected from various media like: twitter, question and answers collections, web searches, news groups, reuters, various scientific documents, public data from different sites and so on. These data are pre-processed with various techniques to remove unwanted information in unstructured data [22] [23] [24].…”

“…For three authors, you may have to improvise. Frequency of Cooccurrence Words 7 [5], [27], [31][32][33][34][35][36] Pseudo documents 6 [22], [23], [36][37][38][39] Word weighting 6 [26], [40][41][42][43][44] Word Embedding 15 [31], [22], [26], [28][ [45][46][47][48][49][50][51][52][53][54][55][56] Sentence Level 1 [57] Hash tags 5 [23], [43], [46], [58], [59]…”

“…Further it eliminates undesired correlations between pseudo-documents and latent topics by applying spike and slab priors. To provide an opportunity for the correlated words to cluster into a similar group, Qiang et al, [22] proposed a novel method of grouping the frequent co-occurrence words and aggregates to pseudo-document using Markov Random Field. Further uses semantic knowledge to extract topics…”

“…Google News - [22], [66], [80] Snippets 8 [8], [29], [36], [44], [45], [49][50][51][52], [68], [74], [75] NIPS - [38] DBLP 6 [31], [37], [62] Yahoo! Answers 11 [38], [45], [77], [82] Online News 7 [26][27][28][29], [31], [32], [37], [47], [62], [68], [81] Baidu Q & A 35 [49], [52], [37], [81][35] [15], [39], [75] Satck Overflow Q & A - [29], [44], [47] Tweets {August to Oct 2008}…”

W2VLDA: Almost unsupervised system for Aspect Based Sentiment Analysis

“…Continuous word embeddings are known for capturing semantic regularities of words (Mikolov et al, 2013a;Collobert and Weston, 2008). Some works have made use of this fact to improve the resulting topics (Das et al, 2015;Nguyen et al, 2015;Qiang et al, 2016), but their objective is to improve the unsupervised modelling of a corpus instead of guiding the model towards a predefined set of topics. There are works that exploit word embeddings in a supervised machine learning setting to perform sentiment analysis (Tang et al, 2014;Giatsoglou et al, 2017).…”