A New Sentence-Based Interpretative Topic Modeling and Automatic Topic Labeling

Abstract

This article presents a new conceptual approach for the interpretative topic modeling problem. It uses sentences as basic units of analysis, instead of words or n-grams, which are commonly used in the standard approaches.The proposed approach’s specifics are using sentence probability evaluations within the text corpus and clustering of sentence embeddings. The topic model estimates discrete distributions of sentence occurrences within topics and discrete distributions of topic occurrence within the text. Our approach provides the possibility of explicit interpretation of topics since sentences, unlike words, are more informative and have complete grammatical and semantic constructions inside. The method for automatic topic labeling is also provided. Contextual embeddings based on the BERT model are used to obtain corresponding sentence embeddings for their subsequent analysis. Moreover, our approach allows big data processing and shows the possibility of utilizing the combination of internal and external knowledge sources in the process of topic modeling. The internal knowledge source is represented by the text corpus itself and often it is a single knowledge source in the traditional topic modeling approaches. The external knowledge source is represented by the BERT, a machine learning model which was preliminarily trained on a huge amount of textual data and is used for generating the context-dependent sentence embeddings.

Ontological Concepts and Logical Listeners for Dysarthric Speech Understanding

“…Each sentence thus has a single topical reference. In order for this to work, it is assumed that sentences with similar themes will also have similar embeddings [30], [31].…”

Advanced Hierarchical Topic Labeling for Short Text

“…Bi-directional Encoder Representation of Transformers (BERT) has been widely utilised as a base framework for various applications. Authors in [44] proposed a BERT-based TM strategy, where they implemented k-means on the sentence BERT embedding matrix to generate topic modeling. The centroid of each cluster has been considered the topic label.…”

“…Multiple strategies have been used to extract the closest possible term, "candidate." Authors in [44] created clusters and assumed the centroid of clusters as closest to all the entities in the cluster, so they considered the centroid of the proposed cluster as the candidate. The authors in [26] used an information extraction method based on BM25 and TF-IDF [27] to extract the candidate terms.…”

Generating Meaningful Topic Descriptions with Sentence Embeddings and LDA

“…Our approach uses sentence-level topic descriptors rather than key words, and we apply a recent sentence encoder that supports multiple languages (Yang et al, 2020). Kozbagarov et al (2021) present another approach to generating interpretable topics by combining sentence embeddings with a topic modeling technique, though they use EM (expectationmaximization) instead of LDA and use averaged BERT word embeddings (Devlin et al, 2019) instead of a pretrained sentence encoder. Like us, they cluster the resulting sentence embeddings and estimate the probability of sentence occurrence within texts, assuming sentences within each cluster as identical.…”