Label Confusion Learning to Enhance Text Classification Models

Abstract

Representing the true label as one-hot vector is the common practice in training text classification models. However, the one-hot representation may not adequately reflect the relation between the instance and labels, as labels are often not completely independent and instances may relate to multiple labels in practice. The inadequate one-hot representations tend to train the model to be over-confident, which may result in arbitrary prediction and model overfitting, especially for confused datasets (datasets with very similar labels) or noisy datasets (datasets with labeling errors). While training models with label smoothing can ease this problem in some degree, it still fails to capture the realistic relation among labels. In this paper, we propose a novel Label Confusion Model (LCM) as an enhancement component to current popular text classification models. LCM can learn label confusion to capture semantic overlap among labels by calculating the similarity between instance and labels during training and generate a better label distribution to replace the original one-hot label vector, thus improving the final classification performance. Extensive experiments on five text classification benchmark datasets reveal the effectiveness of LCM for several widely used deep learning classification models. Further experiments also verify that LCM is especially helpful for confused or noisy datasets and superior to the label smoothing method.

Concept-Based Label Distribution Learning for Text Classification

“…In addition, we tested the accuracy of the model under different dropout rates. Figures 8,9,10,11 show that the optimal accuracy occurs when using Adam with dropout rate= 0.15. We add the dropout between the Bert Layer and the BiL-STM.…”

“…Using hard one-hot label representation may cause model overconfidence, leading to arbitrary predictions while also making it difficult to distinguish some label confusion [5][6][7][8][9][10]. Recently, researchers have focused on soft label representation to solve the above problems, such as label smoothing, label embedding, and label distribution learning.…”

“…This is because we assume a binary relationship between an instance and the labels and assume that the labels are independent of each other. However, labels are rarely completely independent, and instances can be associated with multiple labels [8][9][10][11]. Therefore, the one-hot vector representation is not sufficient to adequately describe the relationship between the instance and the labels.…”

Label Semantic Knowledge Distillation for Unbiased Scene Graph Generation

“…However, such soft labels generated merely by adding noise cannot reflect the correlations between each subjectobject instance and multiple predicate categories. Label confusion (LC) method [60] is proposed in text classification task. It trains a label encoder to learn label representations and calculates the similarity between instances and labels to generate a better label distribution for training.…”

“…It trains a label encoder to learn label representations and calculates the similarity between instances and labels to generate a better label distribution for training. Among these solutions, [60] is the closest to our approach, but we do not need to design additional network encoders to capture correlations between instances and multiple predicates. In addition, some semi-supervised SGG methods also assign soft labels to unannotated samples [61], [62].…”