A Survey of Text Classification With Transformers: How Wide? How Large? How Long? How Accurate? How Expensive? How Safe?

Abstract

Text classification is a basic task in natural language processing (NLP) with applications from sentiment analysis to question-answering with chat bots. In recent years, transformer-based models have emerged as the prevailing framework in NLP, demonstrating excellent results across many benchmarks. This paper recommends an expanded taxonomy of applications and provides a review of the performance of different models across these applications. The use of traditional research techniques plus co-citation and bibliographic coupling provides a comprehensive view of the current and past research in this area. The study begins by providing an overview of the history of transformer-based models with an emphasis on recent large language models (LLM). Next, uni-modal (text only) inputs and the emerging area of multi-modal classification are discussed to provide a comparison of current and emerging research in this area. Gaps are highlighted in the use of multi-modal text/numeric/columnar data and recommendations for future research are provided. Finally, the length of text input variables (tokens) is reviewed to explore the evolution from short-text to longer document applications. Furthermore, the accuracy on 358 datasets across 20 applications is reviewed and unexpected results emerge which show that LLMs are not always the most accurate or least expensive option. In addition to model performance, the safety implications of transformer-based models are reviewed, and a summary of issues related to ethics, bias, social implications, and copyright are explored.

Application of Natural Language Processing and Genetic Algorithm to Fine-Tune Hyperparameters of Classifiers for Economic Activities Analysis

“…Text classification in NLP is rapidly evolving, driven by transformer-based models like LLMs. Fields et al [66] surveys text classification techniques across diverse applications, proposing an expanded taxonomy to include multimodal classification. It evaluates model accuracy, discusses the ethical implications, and emphasizes the importance of a nuanced understanding and holistic deployment in real-world scenarios.…”