Automatic Evaluation of Local Topic Quality

Abstract

Topic models are typically evaluated with respect to the global topic distributions that they generate, using metrics such as coherence, but without regard to local (token-level) topic assignments. Token-level assignments are important for downstream tasks such as classification. Even recent models, which aim to improve the quality of these token-level topic assignments, have been evaluated only with respect to global metrics. We propose a task designed to elicit human judgments of tokenlevel topic assignments. We use a variety of topic model types and parameters and discover that global metrics agree poorly with human assignments.Since human evaluation is expensive we propose a variety of automated metrics to evaluate topic models at a local level. Finally, we correlate our proposed metrics with human judgments from the task on several datasets. We show that an evaluation based on the percent of topic switches correlates most strongly with human judgment of local topic quality. We suggest that this new metric, which we call consistency, be adopted alongside global metrics such as topic coherence when evaluating new topic models.

Topic Model or Topic Twaddle? Re-evaluating Semantic Interpretability Measures

“…Topic model interpretability is a nebulous concept (Lipton, 2018) related to other topic model qualities, but without an agreed-upon definition. Measures of semantic coherence influence how easily understood the top-N T ws are (Morstatter and Liu, 2017;Lund et al, 2019;Newman et al, 2010a;Lau et al, 2014b). This is also referred to as topic understandability (Röder et al, 2015;Aletras et al, 2015).…”

“…Morstatter and Liu (2017) presented interpretability from the perspective of both coherence and consensus, where consensus is a measure of annotator agreement about a topics' representation in its T dc . Alignment is how representative a topic is of its T dc and is another understanding of interpretability (Ando and Lee, 2001;Chang et al, 2009;Mimno et al, 2011;Bhatia et al, 2017;Alokaili et al, 2019;Morstatter and Liu, 2017;Lund et al, 2019). However, the probabilistic nature of topic models impede this measure.…”

“…Following the seminal work of Chang et al (2009), the development of coherence measures and the human evaluation tasks that guide their design has been actively pursued (Newman et al, 2010a;Bhatia et al, 2017Bhatia et al, , 2018Morstatter and Liu, 2017;Lau and Baldwin, 2016;Lund et al, 2019;Alokaili et al, 2019). Newman et al (2010a) showed that human ratings of topic coherence (observed coherence) correlated with their coherence measure when the aggregate Pointwise Mutual Information (PMI) pairwise scores were calculated over the top-N T ws .…”

“…Similarly, Ramrakhiyani et al (2017) made use of the same datasets and evaluations and presented a coherence measure which is approximated with the size of the largest cluster produced from embed-dings of the top-N T ws . Human evaluation tasks have also been created to measure how representative a topic model is of the underlying T dc (Chang et al, 2009;Bhatia et al, 2017;Morstatter and Liu, 2017;Alokaili et al, 2019;Lund et al, 2019).…”

“…The alignment of T ws to T dc is an expected property of a "good" topic (Chang et al, 2009), which human evaluation tasks have been developed to assess. Typically these tasks ask annotators to choose the most and/or least aligned T ws to a given document (Morstatter and Liu, 2017;Lund et al, 2019;Alokaili et al, 2019;Bhatia et al, 2018), identify an intruder topic (Chang et al, 2009;Morstatter and Liu, 2017), rate their confidence in a topicdocument pair (Bhatia et al, 2017), or select appropriate documents given a category label (Aletras et al, 2017). However, none of these methods address the need for the topic document-collection to be evaluated and labeled.…”