Improving BERT-Based Text Classification With Auxiliary Sentence and Domain Knowledge

Abstract

General language model BERT pre-trained on cross-domain text corpus, BookCorpus and Wikipedia, achieves excellent performance on a couple of natural language processing tasks through the way of fine-tuning in the downstream tasks. But it still lacks of task-specific knowledge and domain-related knowledge for further improving the performance of BERT model and more detailed fine-tuning strategy analyses are necessary. To address these problem, a BERT-based text classification model BERT4TC is proposed via constructing auxiliary sentence to turn the classification task into a binary sentence-pair one, aiming to address the limited training data problem and task-awareness problem. The architecture and implementation details of BERT4TC are also presented, as well as a post-training approach for addressing the domain challenge of BERT. Finally, extensive experiments are conducted on seven public widelystudied datasets for analyzing the fine-tuning strategies from the perspectives of learning rate, sequence length and hidden state vector selection. After that, BERT4TC models with different auxiliary sentences and post-training objectives are compared and analyzed in depth. The experiment results show that BERT4TC with suitable auxiliary sentence significantly outperforms both typical feature-based methods and finetuning methods, and achieves new state-of-the-art performance on multi-class classification datasets. For binary sentiment classification datasets, our BERT4TC post-trained with suitable domain-related corpus also achieves better results compared with original BERT model. INDEX TERMS Natural language processing, text classification, bidirectional encoder representations from transformer, neural networks, language model.

A Fine-Tuned BERT-Based Transfer Learning Approach for Text Classification

“…They have experienced that transfer learning models can perform better than other state-of-the-art methods in NLP. BERT is trained on BookCorpus, text corpus, and Wikipedia which can give overwhelming results in some areas of natural language processing but it still needs to be improved [ 32 ]. It somewhere lacks domain-related and task-related knowledge.…”

A Comparison of Pre-Trained Language Models for Multi-Class Text Classification in the Financial Domain

“…Sun et al [41] make detailed experiments on BERT and suggest several techniques to improve the results on text classification task. Yu et al [50] propose a BERT-based model for text classification to utilize more task-specific knowledge and achieve better results on multi-classification task. Yeung [49] inserts legal domain vocabulary to BERT, reports no improvement and explains their findings by the high overlap between vocabularies.…”

Sentence-Level Aspect-Based Sentiment Analysis for Classifying Adverse Drug Reactions (ADRs) Using Hybrid Ontology-XLNet Transfer Learning

“…Likewise, the precision represents the ratio of correctly predicted positive labels through the proposed approach to the total predicted positive labels. Consequently, the F-measure is calculated as the harmonic mean of precision and recall to show the cumulative effect of both measures as shown in equation (12).…”

“…The BERT model is designed to pre-train deep bidirectional representations of unlabeled text by co-conditioning both left and right context in all layers. A different contextual embeddings is produced by BERT according to the input sentence [12]. Nevertheless, BERT corrupts the input with masks, suffers a discrepancy between pre-training and fine-tuning, and ignores the interdependency between masked positions, thus leading to the loss of important information [13]- [15].…”