A Hybrid BERT Model That Incorporates Label Semantics via Adjustive Attention for Multi-Label Text Classification

Cai, Linkun; Shen, Yu; Liu, Tao; Zhang, Kunli

doi:10.1109/access.2020.3017382

Cited by 67 publications

(25 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…(2018) proposed BERT, an innovative deep neural network model that includes the pre‐training of deep bidirectional representations from unlabelled text and fine‐tuning with one additional output layer. BERT has achieved the best performance in text classification (Cai et al., 2020; Dong et al., 2020). Thus, this study adopted BERT to automatically classify online discussion transcripts into three interaction types: cognitive, metacognitive and off‐topic.…”

Section: Methodsmentioning

confidence: 99%

Effects of a learning analytics‐based real‐time feedback approach on knowledge elaboration, knowledge convergence, interactive relationships and group performance in CSCL

Zheng

Niu

Zhong

2021

Brit J Educational Tech

View full text Add to dashboard Cite

Learning analytics (LA) has been widely adopted in research on education. However, most studies in the area have conducted LA after computer‐supported collaborative learning (CSCL) activities rather than during CSCL. To address this problem, this study proposed a LA‐based real‐time feedback approach based on a deep neural network model to improve CSCL performance. In total, 72 university students participated in the study and were randomly assigned to an experimental or control group. The students in the experimental group learned with the LA‐based real‐time feedback approach, whereas the students in the control group learned with the conventional online collaborative learning approach. To analyse the data, both quantitative and qualitative methods were adopted. The results indicated that the LA‐based real‐time feedback approach significantly promoted knowledge convergence, knowledge elaboration, interactive relationships and group performance. The interview results also confirmed the effectiveness of the proposed approach. What is already known regarding this topic Learning analytics (LA) has been widely used in education. Most studies in the area have presented LA results only after collaborative learning and have lacked real‐time analysis and feedback. What this paper adds A LA‐based real‐time feedback approach was proposed and validated in the computer‐supported collaborative learning (CSCL) context. The experimental results indicated that the LA‐based real‐time feedback approach significantly promoted knowledge elaboration, knowledge convergence, interactive relationships and group performance. Implications for practice and/or policy To shed light on progress in CSCL, real‐time LA are recommended. Deep neural network models, such as bidirectional encoder representations from transformers, can be adopted to automatically analyse online discussion transcripts. Real‐time feedback based on LA results can promote CSCL performance.

show abstract

Section: Methodsmentioning

confidence: 99%

Effects of a learning analytics‐based real‐time feedback approach on knowledge elaboration, knowledge convergence, interactive relationships and group performance in CSCL

Zheng

Niu

Zhong

2021

Brit J Educational Tech

View full text Add to dashboard Cite

show abstract

“…A hybrid BERT model was proposed for multi-label text classification [19]. This model works based on four sub-task; first context-aware representation was developed using the word embedding technique (pre-trained BERT).…”

Section: Indonesian J Elec Eng and Comp Sci Issn: 2502-4752 mentioning

confidence: 99%

“…GCN collects the values of all neighboring nodes to evaluate the current node. ReLU activation function used here [19], [27].…”

mentioning

confidence: 99%

Analyzing semantic similarity amongst textual documents to suggest near duplicates

Devarajan

Subramanian

2022

IJEECS

View full text Add to dashboard Cite

Data deduplication techniques removing repeated or redundant data from the storage. In recent days, more data has been generated and stored in the storage environment. More redundant and semantically similar content of the data occupied in the storage environment due to this storage efficiency will be reduced and cost of the storage will be high. To overcome this problem, we proposed a method hybrid bidirectional encoder representation from transformers for text semantics using graph convolutional network hybrid bidirectional encoder representation from transformers (BERT) model for text semantics (HBTSG) word embedding-based deep learning model to identify near duplicates based on the semantic relationship between text documents. In this paper we hybridize the concepts of chunking and semantic analysis. The chunking process is carried out to split the documents into blocks. Next stage we identify the semantic relationship between documents using word embedding techniques. It combines the advantages of the chunking, feature extraction, and semantic relations to provide better results.

show abstract

“…DUTSD has the best performance among the three separate dictionaries of sentiment. BERT representation technique is used for ABSA [13,14]. Many previous methodologies treated labels as symbols without semantics and ignored the relation among labels, which caused information loss.…”

Section: Related Workmentioning

confidence: 99%

“…Many previous methodologies treated labels as symbols without semantics and ignored the relation among labels, which caused information loss. This problem is handled by Cai et al [13]. In this approach, the hybrid BERT model incorporates label semantics via ajustive attention, which searches and identifies semantic dependencies of label space and text space simultaneously.…”

Section: Related Workmentioning

confidence: 99%

A Multilabel Classifier for Text Classification and Enhanced BERT System

Bhamare¹,

Prabhu²

2021

RIA

View full text Add to dashboard Cite

Now-a-day, a vast variety of reviews are published on the web. As a result, an automated system to analyze and extract knowledge from such textual data is needed. Sentiment analysis is a well-known sub-area in Natural Language Processing (NLP). In earlier research, sentiments were determined without considering the aspects specified in a review instance. Aspect-based sentiment analysis (ABSA) has caught the attention of researchers. Many existing systems consider ABSA as a single label classification problem. This drawback is handled in this study by proposing three approaches that use multilabel classifiers for classification. In the first approach, the performance of a model with hybrid features is analyzed using the multilabel classifier. The hybrid feature set includes word dependency rule-based features and unigram features selected using the proposed two-phase weighted correlation feature selection (WCFS) approach. In the second and third approaches Bidirectional Encoder Representation from Transformers (BERT) language model is used. In the second approach, a BERT system is enhanced by applying max pooling on target terms which specify an aspect of a review instance and a multibit label is given as input to the BERT system. In the third approach, the basic BERT system is used for word embedding only and classification is done using multilabel classifiers. In all approaches, the label used for all training instances specifies aspects with its sentiments. The experimentation shows that the results gained using the system proposed in the first approach are comparable to the results gained using the BERT system. The experimental results depict that the Enhanced BERT system gives better results compared to the existing systems.

show abstract

A Hybrid BERT Model That Incorporates Label Semantics via Adjustive Attention for Multi-Label Text Classification

Cited by 67 publications

References 28 publications

Effects of a learning analytics‐based real‐time feedback approach on knowledge elaboration, knowledge convergence, interactive relationships and group performance in CSCL

Effects of a learning analytics‐based real‐time feedback approach on knowledge elaboration, knowledge convergence, interactive relationships and group performance in CSCL

Analyzing semantic similarity amongst textual documents to suggest near duplicates

A Multilabel Classifier for Text Classification and Enhanced BERT System

Contact Info

Product

Resources

About