Measuring Topic Coherence through Optimal Word Buckets

Ramrakhiyani, Nitin; Pawar, Sachin; Hingmire, Swapnil; Palshikar, Girish Keshav

doi:10.18653/v1/e17-2070

Cited by 7 publications

(9 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Instead of calculating pairwise word similarities, Rosner et al (2014) proposed the partitioning of the set of top topic words into subsets and averaged a "confirmation measure" based on conditional probability over the subset pairs. Ramrakhiyani et al (2017) proposed a coherence measure based on clustering the embeddings of top topic words and approximating the coherence with the size of the largest cluster.…”

Section: Topic Coherencementioning

confidence: 99%

“…More specifically, Chang et al showed that models that fare better in predictive perplexity often have less interpretable topics, suggesting that evaluation should consider the internal representation of topic models and aim to quantify their interpretability. The idea soon gave rise to a new family of methods (Newman et al, 2010;Lau et al, 2014;Röder et al, 2015;Ramrakhiyani et al, 2017) that evaluate the semantic interpretability by measuring the topic coherence. These methods assume that topic coherence correlates with the coherence of the words assigned to that topic and thus quantify topic coherence as the coherence of top-ranked topic words.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Document-based topic coherence measures for news media text

Korenčić

Ristov

Šnajder

2018

Expert Systems with Applications

View full text Add to dashboard Cite

With the widespread and increased consumption of online news, there is a rising need for automated analysis of news text. Topic models have proven to be useful tools for unsupervised discovery of topics from large amounts of text, including news media texts. Topics produced by a topic model are often represented as probability-weighted word lists, and it is expected that these bear correspondence to semantic topics-semantic concepts representable by a topic model. However, because the quality of topics varies and not all topics correspond to semantic topics in practice, much research effort has been devoted to automated evaluation of topic models. One class of popular and effective methods focuses on topic coherence as a measure of a topic's semantic interpretability and its correspondence to a semantic topic. Existing topic coherence methods calculate the coherence score based on the semantic similarity of topic-related words. However, news media texts revolve around specific news stories, giving rise to many contingent and transient topics for which topic-related words tend to be semantically unrelated. Consequently, the coherence of many news media topics is not amenable to detection via state-of-art word-based coherence measures. In this paper, we propose a novel class of topic coherence methods that estimate topic coherence based on topic documents rather than topic words. We evaluate the proposed methods on two datasets containing topics manually labeled for document-based coherence, derived from US and Croatian news text corpora. Our best-performing document-based coherence measure achieves an AUC score above 0.8, substantially outperforming a strong baseline method and state-of-art word-based coherence methods. We also demonstrate that there may be benefit in combining word-and document-based coherence measures. Lastly, we demonstrate the usefulness of document-based coherence measures for automated topic discovery from news media texts.

show abstract

Section: Topic Coherencementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Document-based topic coherence measures for news media text

Korenčić

Ristov

Šnajder

2018

Expert Systems with Applications

View full text Add to dashboard Cite

show abstract

“…Similarly, Ramrakhiyani et al (2017) made use of the same datasets and evaluations and presented a coherence measure which is approximated with the size of the largest cluster produced from embed-dings of the top-N T ws . Human evaluation tasks have also been created to measure how representative a topic model is of the underlying T dc (Chang et al, 2009;Bhatia et al, 2017;Morstatter and Liu, 2017;Alokaili et al, 2019;Lund et al, 2019).…”

Section: Related Workmentioning

confidence: 99%

Topic Model or Topic Twaddle? Re-evaluating Semantic Interpretability Measures

Doogan¹,

Buntine²

2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

When developing topic models, a critical question that should be asked is: How well will this model work in an applied setting? Because standard performance evaluation of topic interpretability uses automated measures modeled on human evaluation tests that are dissimilar to applied usage, these models' generalizability remains in question. In this paper, we probe the issue of validity in topic model evaluation and assess how informative coherence measures are for specialized collections used in an applied setting. Informed by the literature, we propose four understandings of interpretability. We evaluate these using a novel experimental framework reflective of varied applied settings, including human evaluations using open labeling, typical of applied research. These evaluations show that for some specialized collections, standard coherence measures may not inform the most appropriate topic model or the optimal number of topics, and current interpretability performance validation methods are challenged as a means to confirm model quality in the absence of ground truth data.

show abstract

“…Coherence measures are commonly based on a score of mutual similarity between top topic-related words, which can be defined using a variety of word representations and similarity measures [9], [11], [42], [63], [73]- [77]. Alternate approaches include clustering of word embeddings [78] and querying search engines with top topic words [9]. In addition to topic coherence measures, alternate approaches to calculating topic quality have been proposed, based on calculating distances between topics and uninformative probability distributions [79], and on aligning model topics with WordNet concepts [80], [81].…”

Section: B Topic Model Evaluationmentioning

confidence: 99%

A Topic Coverage Approach to Evaluation of Topic Models

et al. 2021

View full text Add to dashboard Cite

Topic models are widely used unsupervised models capable of learning topics -weighted lists of words and documents -from large collections of text documents. When topic models are used for discovery of topics in text collections, a question that arises naturally is how well the model-induced topics correspond to topics of interest to the analyst. In this paper we revisit and extend a so far neglected approach to topic model evaluation based on measuring topic coverage -computationally matching model topics with a set of reference topics that models are expected to uncover. The approach is well suited for analyzing models' performance in topic discovery and for large-scale analysis of both topic models and measures of model quality. We propose new measures of coverage and evaluate, in a series of experiments, different types of topic models on two distinct text domains for which interest for topic discovery exists. The experiments include evaluation of model quality, analysis of coverage of distinct topic categories, and the analysis of the relationship between coverage and other methods of topic model evaluation. The paper contributes a new supervised measure of coverage, and the first unsupervised measure of coverage. The supervised measure achieves topic matching accuracy close to human agreement. The unsupervised measure correlates highly with the supervised one (Spearman's ρ ≥ 0.95). Other contributions include insights into both topic models and different methods of model evaluation, and the datasets and code for facilitating future research on topic coverage.

show abstract

Measuring Topic Coherence through Optimal Word Buckets

Cited by 7 publications

References 14 publications

Document-based topic coherence measures for news media text

Document-based topic coherence measures for news media text

Topic Model or Topic Twaddle? Re-evaluating Semantic Interpretability Measures

A Topic Coverage Approach to Evaluation of Topic Models

Contact Info

Product

Resources

About