Bag-of-Words vs. Sequence vs. Graph vs. Hierarchy for Single- and Multi-Label Text Classification

Abstract

Graph neural networks have triggered a resurgence of graph-based text classification methods, defining today's state of the art. We show that a simple multi-layer perceptron (MLP) using a Bag of Words (BoW) outperforms the recent graph-based models TextGCN and Hete-GCN in an inductive text classification setting and is comparable with HyperGAT in singlelabel classification. We also run our own experiments on multi-label classification, where the simple MLP outperforms the recent sequentialbased gMLP and aMLP models. Moreover, we fine-tune a sequence-based BERT and a lightweight DistilBERT model, which both outperform all models on both single-label and multi-label settings in most datasets. These results question the importance of synthetic graphs used in modern text classifiers. In terms of parameters, DistilBERT is still twice as large as our BoW-based wide MLP, while graph-based models like TextGCN require setting up an O(N 2 ) graph, where N is the vocabulary plus corpus size.

Prediction of People Sentiments on Twitter Using Machine Learning Classifiers During Russian-Ukrainian Conflict

“…The MLP is an arti cial neural network architecture, which is probably the most widely used for classi cation and regression today [39]. MLPs are feed-forward neural networks usually made up of several layers of nodes that only connect in one direction and are usually trained by backpropagation [40].…”

Vocational Education as a Tool for Socio-economic Development: A Case Study of Wuling Ethnic Areas

“…The "Chongming Rose" and "Jufeng Grape" won the gold medal at the 5th Hunan Provincial "Hunan Portugal Green Tasting Cup" grape tasting event in 2018. Economic crops are harvested independently, ordered through phone and WeChat, and all grapes in the Wuling ethnic area are sold out without leaving the village (Diera, et al, 2022).…”