A Topic Coverage Approach to Evaluation of Topic Models

Abstract

Topic models are widely used unsupervised models capable of learning topics -weighted lists of words and documents -from large collections of text documents. When topic models are used for discovery of topics in text collections, a question that arises naturally is how well the model-induced topics correspond to topics of interest to the analyst. In this paper we revisit and extend a so far neglected approach to topic model evaluation based on measuring topic coverage -computationally matching model topics with a set of reference topics that models are expected to uncover. The approach is well suited for analyzing models' performance in topic discovery and for large-scale analysis of both topic models and measures of model quality. We propose new measures of coverage and evaluate, in a series of experiments, different types of topic models on two distinct text domains for which interest for topic discovery exists. The experiments include evaluation of model quality, analysis of coverage of distinct topic categories, and the analysis of the relationship between coverage and other methods of topic model evaluation. The paper contributes a new supervised measure of coverage, and the first unsupervised measure of coverage. The supervised measure achieves topic matching accuracy close to human agreement. The unsupervised measure correlates highly with the supervised one (Spearman's ρ ≥ 0.95). Other contributions include insights into both topic models and different methods of model evaluation, and the datasets and code for facilitating future research on topic coverage.

A data-driven analysis to determine the optimal number of topics 'K' for latent Dirichlet allocation model

“…The quality of topics generated by topic models can be assessed using evaluation metrics defined to quantify various topics, such as correlation, similarity, saliency, and relevance. Based on extensive research, various metrics and techniques have been devised to evaluate the topics produced by topic models and determine the optimal number of topics over the last two decades [36], [37].…”

How Well Environmental Design Is and Can Be Suited to People with Autism Spectrum Disorder (ASD): A Natural Language Processing Analysis

“…To obtain the prominent topics or nent words within the inspected documents, topic modeling was implemented t the Orange TM toolkit. It was used to analyze and find the frequency of words in cles' abstracts, thus providing an indication about relevance to the inspected doma lowing Korenčcić et al [41], the LDA unsupervised learning algorithm, which is the most popular topic modeling methods, was used. The algorithm is designed lyze a large volume of unlabeled text.…”

Trends and Challenges Regarding Cyber Risk Mitigation by CISOs—A Systematic Literature and Experts’ Opinion Review Based on Text Analytics

“…The topic modeling procedure was used to analyze two types of document corpora: scientific papers and expert opinion columns. The analysis was performed on both repositories in the same manner: following Korenčić et al [23], the latent Dirichlet allocation (LDA) unsupervised learning algorithm was used to analyze a large volume of unlabeled text through the evaluation of clusters of words that frequently occur together [24,25], under the assumption that similar words appearing in similar contexts represent the same topic. Subsequently, the scientific corpus and the expert opinion columns' content were analyzed using the following methods: (1) A search for paragraphs containing words provided by the Merriam-Webster online thesaurus lexicon (https://www.merriam-webster.com/thesaurus, accessed on 29 December 2021).…”