Making LLMs Worth Every Penny: Resource-Limited Text Classification in Banking

Abstract

Standard Full-Data classifiers in NLP demand thousands of labeled examples, which is impractical in data-limited domains. Fewshot methods offer an alternative, utilizing contrastive learning techniques that can be effective with as little as 20 examples per class. Similarly, Large Language Models (LLMs) like GPT-4 can perform effectively with just 1-5 examples per class. However, the performance-cost trade-offs of these methods remain underexplored, a critical concern for budget-limited organizations. Our work addresses this gap by studying the aforementioned approaches over the Banking77 financial intent detection dataset, including the evaluation of cutting-edge LLMs by OpenAI, Cohere, and Anthropic in a comprehensive set of few-shot scenarios. We complete the picture with two additional methods: first, a cost-effective querying method for LLMs based on retrieval-augmented generation (RAG), able to reduce operational costs multiple times compared to classic few-shot approaches, and second, a data augmentation method using GPT-4, able to improve performance in data-limited scenarios. Finally, to inspire future research, we provide a human expert's curated subset of Banking77, along with extensive error analysis. CCS CONCEPTS• Computing methodologies → Natural language processing.

Advancing entity recognition in biomedicine via instruction tuning of large language models

“…Recently, generative pre-trained transformer (GPT) models with billion scale parameters such as GPT-3.5 ( https://platform.openai.com/docs/models/gpt-3-5 ) and GPT-4 ( Achiam et al 2023 ) have achieved state-of-the-art results on a wide range of NLP tasks, including text classification, summarization, question answering, and translation especially in zero or few-shot setting, which may potentially address the limitations mentioned above ( Gilardi et al 2023 , Gilson et al 2023 , Hendy et al 2023 , Loukas et al , 2023 , Manakhimova et al 2023 , Wang et al 2023a , 2023b ). These large language models (LLMs) require huge computational resources to train and are often trained on massive amounts of data including proprietary datasets.…”