tBERT: Topic Models and BERT Joining Forces for Semantic Similarity Detection

Peinelt, Nicole; Nguyen, Dong; Liakata, Maria

doi:10.18653/v1/2020.acl-main.630

Cited by 119 publications

(46 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this study, we characterized the responses via feature extraction by employing a combination of the latent Dirichlet allocation model (LDA) and semantic representation transformers called Bidirectional Encoder Representations from Transformers (BERT), then grouped similar responses using k-means clustering. Based on studies demonstrating that this combination improved performance on numerous language analysis tasks (Rangrej et al, 2011;Peinelt et al, 2020;Xie et al, 2020), this approach allowed us to better reveal the most common types of silver linings in participants' responses.…”

Section: (B) Sentiment Analysismentioning

confidence: 99%

Identifying Silver Linings During the Pandemic Through Natural Language Processing

et al. 2021

View full text Add to dashboard Cite

COVID-19 has presented an unprecedented challenge to human welfare. Indeed, we have witnessed people experiencing a rise of depression, acute stress disorder, and worsening levels of subclinical psychological distress. Finding ways to support individuals' mental health has been particularly difficult during this pandemic. An opportunity for intervention to protect individuals' health & well-being is to identify the existing sources of consolation and hope that have helped people persevere through the early days of the pandemic. In this paper, we identified positive aspects, or “silver linings,” that people experienced during the COVID-19 crisis using computational natural language processing methods and qualitative thematic content analysis. These silver linings revealed sources of strength that included finding a sense of community, closeness, gratitude, and a belief that the pandemic may spur positive social change. People's abilities to engage in benefit-finding and leverage protective factors can be bolstered and reinforced by public health policy to improve society's resilience to the distress of this pandemic and potential future health crises.

show abstract

Section: (B) Sentiment Analysismentioning

confidence: 99%

Identifying Silver Linings During the Pandemic Through Natural Language Processing

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Thereafter, we refer to this enhanced modeling method of PBERT as TPBERT, whose model structure is schematically depicted in Figure 2. Our TPBERT bears some resemblance to a recently proposed BERT-based model for semantic similarity detection [29].…”

Section: Incorporation Of Task-specific Topic Information Into Pbertmentioning

confidence: 80%

Innovative Bert-Based Reranking Language Models for Speech Recognition

Chiu

Chen

2021

2021 IEEE Spoken Language Technology Workshop (SLT)

View full text Add to dashboard Cite

More recently, Bidirectional Encoder Representations from Transformers (BERT) was proposed and has achieved impressive success on many natural language processing (NLP) tasks such as question answering and language understanding, due mainly to its effective pre-training then fine-tuning paradigm as well as strong local contextual modeling ability. In view of the above, this paper presents a novel instantiation of the BERT-based contextualized language models (LMs) for use in reranking of N-best hypotheses produced by automatic speech recognition (ASR). To this end, we frame N-best hypothesis reranking with BERT as a prediction problem, which aims to predict the oracle hypothesis that has the lowest word error rate (WER) given the N-best hypotheses (denoted by PBERT). In particular, we also explore to capitalize on task-specific global topic information in an unsupervised manner to assist PBERT in N-best hypothesis reranking (denoted by TPBERT). Extensive experiments conducted on the AMI benchmark corpus demonstrate the effectiveness and feasibility of our methods in comparison to the conventional autoregressive models like the recurrent neural network (RNN) and a recently proposed method that employed BERT to compute pseudo-log-likelihood (PLL) scores for N-best hypothesis reranking.

show abstract

“…Due to the large number of parameters required by the BERT model and its advantages in local semantic representation, this model cannot represent the macro-domain information of the input document other than its own semantics. Inspired by reference [8] and reference [9], we propose a document semantic acquisition method based on tALBERT. We obtain the topic information at the word level and document level through an LDA topic model, obtain a semantic representation at the wordlevel and document-level through ALBERT, and fuse the above information through a concatenation mechanism to represent the document.…”

Section: Model Frameworkmentioning

confidence: 99%

“…In 2020, Lan et al [8] proposed "A Lite BERT" (ALBERT) model, which greatly simplifies the number of required parameters. Peinelt et al [9] combined a topic model with a BERT model for the task of semantic similarity detection. We have reason to use ALBERT and topic models to extract important information of different granularities form documents to further improve the effect of multi-label text classification.…”

Section: Introductionmentioning

confidence: 99%

Research on Multi-label Text Classification Method Based on tALBERT-CNN

Liu

Pang²,

Li³

et al. 2021

Int J Comput Intell Syst

View full text Add to dashboard Cite

Single-label classification technology has difficulty meeting the needs of text classification, and multi-label text classification has become an important research issue in natural language processing (NLP). Extracting semantic features from different levels and granularities of text is a basic and key task in multi-label text classification research. A topic model is an effective method for the automatic organization and induction of text information. It can reveal the latent semantics of documents and analyze the topics contained in massive information. Therefore, this paper proposes a multi-label text classification method based on tALBERT-CNN: an LDA topic model and ALBERT model are used to obtain the topic vector and semantic context vector of each word (document), a certain fusion mechanism is adopted to obtain in-depth topic and semantic representations of the document, and the multi-label features of the text are extracted through the TextCNN model to train a multi-label classifier. The experimental results obtained on standard datasets show that the proposed method can extract multi-label features from documents, and its performance is better than that of the existing state-of-the-art multi-label text classification algorithms.

show abstract

tBERT: Topic Models and BERT Joining Forces for Semantic Similarity Detection

Cited by 119 publications

References 28 publications

Identifying Silver Linings During the Pandemic Through Natural Language Processing

Identifying Silver Linings During the Pandemic Through Natural Language Processing

Innovative Bert-Based Reranking Language Models for Speech Recognition

Research on Multi-label Text Classification Method Based on tALBERT-CNN

Contact Info

Product

Resources

About