Topic Model or Topic Twaddle? Re-evaluating Semantic Interpretability Measures

Abstract

When developing topic models, a critical question that should be asked is: How well will this model work in an applied setting? Because standard performance evaluation of topic interpretability uses automated measures modeled on human evaluation tests that are dissimilar to applied usage, these models' generalizability remains in question. In this paper, we probe the issue of validity in topic model evaluation and assess how informative coherence measures are for specialized collections used in an applied setting. Informed by the literature, we propose four understandings of interpretability. We evaluate these using a novel experimental framework reflective of varied applied settings, including human evaluations using open labeling, typical of applied research. These evaluations show that for some specialized collections, standard coherence measures may not inform the most appropriate topic model or the optimal number of topics, and current interpretability performance validation methods are challenged as a means to confirm model quality in the absence of ground truth data.

Using ICD-9 diagnostic codes for external validation of topic models derived from primary care electronic medical record clinical text data

“…This type of post-hoc expert informed inspection builds faith in the face validity of the learned model; however, it is criticized for lacking rigor compared with alternative approaches based on quantitative evaluation metrics. 12,13…”

“…For many of the learned thematic vectors we observed a strong degree of semantic overlap between the words/tokens (summarizing fitted topical vectors) and the ICD-9 diagnostic codes identified as being most strongly associated with the thematic vector. Subjectively, the following topical vectors demonstrated reasonable convergent/discriminant validity: (21,27,13,18,26,15,47,48,11,50,41,23,39,3,14,32,38,4,5,8,46,25,7,9,45). Below, we identified a subset of thematic vectors for which the words/tokens loading strongly on topical basis appeared semantically associated with assigned primary diagnostic codes, suggesting they may be measuring the same latent construct:…”

“…This type of post-hoc expert informed inspection builds faith in the face validity of the learned model; however, it is criticized for lacking rigor compared with alternative approaches based on quantitative evaluation metrics. 12,13 Internal validation is another common approach for validating a fitted topic model (where model learned quantities are inspected with respect to internal robustness, stability, predictive, geometric or semantic properties). Several sensible internal validation schemes exist, for demonstrating topic model validity.…”

A Common Derivation for Parsing and Generation with Expectation-Based Minimalist Grammars (e-MGs)

“…On the other hand, the topics discovered by ETM are more stable but have a lower coherence on average. As already observed in previous work (Al-Sumait et al, 2009;Doogan and Buntine, 2021), obtaining junk or mixed topics is common in topic models and this problem can be addressed by filtering out the topics that are less relevant.…”

A systematic review of the use of topic models for short text social media analysis

“… 2018 ; Hoyle et al. 2020 ; Doogan and Buntine 2021 ). Recent approaches to modelling short text datasets include the use of auxiliary metadata (Zhao et al.…”