Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning

Wu, Xiaobao; Luu, Anh Tuan; Dong, Xinshuai

doi:10.18653/v1/2022.emnlp-main.176

Cited by 9 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This could be achieved by training a decoder for summarization, generating a summarization for each topic, and subsequently concatenating them. This framework can also be extended to hierarchical topic modeling (Chen et al, 2023;Shahid et al, 2023;Eshima and Mochihashi, 2023), mitigate data sparsity for short text topic modeling (Wu et al, 2022), generate topic-relevant and coherent long texts (Yang et al, 2022), and construct a network of topics together with meaningful relationships between them (Byrne et al, 2022).…”

Section: Discussionmentioning

confidence: 99%

vONTSS: vMF based semi-supervised neural topic modeling with optimal transport

Xu¹,

Jiang²,

Rao³

et al. 2023

Findings of the Association for Computational Linguistics: ACL 2023

View full text Add to dashboard Cite

Language model based methods are powerful techniques for text classification. However, the models have several shortcomings. (1) It is difficult to integrate human knowledge such as keywords. (2) It needs a lot of resources to train the models.(3) It relied on large text data to pretrain. In this paper, we propose Semi-Supervised vMF Neural Topic Modeling (S2vNTM) to overcome these difficulties. S2vNTM takes a few seed keywords as input for topics. S2vNTM leverages the pattern of keywords to identify potential topics, as well as optimize the quality of topics' keywords sets. Across a variety of datasets, S2vNTM outperforms existing semi-supervised topic modeling methods in classification accuracy with limited keywords provided. S2vNTM is at least twice as fast as baselines.

show abstract

Section: Discussionmentioning

confidence: 99%

vONTSS: vMF based semi-supervised neural topic modeling with optimal transport

Xu¹,

Jiang²,

Rao³

et al. 2023

Findings of the Association for Computational Linguistics: ACL 2023

View full text Add to dashboard Cite

show abstract

“…To enhance text representations through self-supervision in an unsupervised environment, some researchers applied contrastive learning to pre-trained language models like BERT (Kim et al, 2021;Yan et al, 2021). Wu et al (2022) proposed a topic-semantic contrastive topic model (TSCTM) to solve the issue of data sparsity.…”

Section: Contrastive Learningmentioning

confidence: 99%

A multi‐label social short text classification method based on contrastive learning and improved ml‐KNN

Tian,

Wang,

Wang

et al. 2024

Expert Systems

View full text Add to dashboard Cite

Short texts on social platforms often have the problems of diverse categories and semantic sparsity, making it challenging to identify the diverse intentions of users. To address this issue, this article proposes a multi‐label social short text classification method (IML‐CL) based on contrastive learning and improved ml‐KNN. First, a contrastive learning approach is employed to train a multi‐label text classification model. This approach improves semantic sparsity by leveraging the knowledge from the existing samples to enrich the feature representation of short texts. Simultaneously, an improved ml‐KNN algorithm is developed to enhance the accuracy of label prediction. This algorithm utilizes a two‐layer nearest neighbor rule and introduces a penalty function and weight optimization. Next, the model generates the feature representation for the test sample and predicts its label. Additionally, the improved ml‐KNN algorithm retrieves neighbors of the test sample and uses their label information for prediction. Finally, the two predictions are combined to obtain the final prediction, which accurately identifies the user's intention. The experimental results demonstrate that, on the dataset constructed in this article, the IML‐CL method effectively boosts the performance of the baseline model.

show abstract

“…propose the contrastive learning on doc-topic distributions where they build positive and negative pairs by sampling salient words of documents. Differently, Wu et al (2022) directly sample positive and negative pairs based on the topic semantics of documents to capture relations among samples. Specifically, they quantize doc-topic distributions following Wu et al (2020b) and then sample documents with the same quantization indices as positive pairs and different indices as negative pairs.…”

Section: Ntms With Contrastive Learningmentioning

confidence: 99%

“…They learn entity vector representations from manually edited knowledge graphs. Based on Wu et al (2020b), Wu et al (2022) further propose a contrastive learning method according to the topic semantics of short texts, which better captures the similarity relations among them. This refines the representations of short texts and thus their doctopic distributions.…”

Section: Short Text Ntmsmentioning

confidence: 99%

A Survey on Neural Topic Models: Methods, Applications, and Challenges

Nguyen

Luu

2023

Preprint

Self Cite

View full text Add to dashboard Cite

Topic models aim to discover latent topics and infer what topics a document contains in an unsupervised fashion. They have been prevalent for decades with wide applications like text analysis. Recently the rise of neural networks has facilitated a new research field---Neural Topic Models (NTMs). Different from conventional models, NTMs directly optimize parameters without model-specific derivations. This endows NTMs with better scalability and flexibility, resulting in significant research attention and plentiful new methods and applications. In this paper, we present a thorough survey on neural topic models carefully and widely. In detail, we systematically organize current NTM methods according to their network structures and introduce the NTMs for various scenarios like short texts and cross-lingual documents. Then we discuss developed popular applications based on NTMs. We finally highlight the challenges of NTMs to motivate future research.

show abstract

Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning

Cited by 9 publications

References 0 publications

vONTSS: vMF based semi-supervised neural topic modeling with optimal transport

vONTSS: vMF based semi-supervised neural topic modeling with optimal transport

A multi‐label social short text classification method based on contrastive learning and improved ml‐KNN

A Survey on Neural Topic Models: Methods, Applications, and Challenges

Contact Info

Product

Resources

About