Azimuth: Systematic Error Analysis for Text Classification

Gauthier-melancon, Gabrielle; Ayala, Orlando Marquez; Brin, Lindsay D.; Tyler, Christopher J.; Branchaud-Charron, Frédéric; Joseph, Marinier,; Grande, Karine; Le, Di

doi:10.18653/v1/2022.emnlp-demos.30

Cited by 2 publications

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since Sahu et al found that data augmentation for tasks with large label volume is likely to not benefit at all (at least when using older text generation models), we adopt a particular prompt template for our instruct-based GPT-4 model. First, we analyze the data using Azimuth [13], an open-source toolkit. We split the 77 labels into ten groups, where each group consists of labels that contain intents with highly semantic overlap, as shown in Azimuth (e.g., Top Up Reverted and Top Up Failed belong to the same group).…”

Section: Llms For Data Generation In Low-resource Settingsmentioning

confidence: 99%

Making LLMs Worth Every Penny: Resource-Limited Text Classification in Banking

Loukas,

Stogiannidis,

Diamantopoulos

et al. 2023

4th ACM International Conference on AI in Finance

View full text Add to dashboard Cite

Standard Full-Data classifiers in NLP demand thousands of labeled examples, which is impractical in data-limited domains. Fewshot methods offer an alternative, utilizing contrastive learning techniques that can be effective with as little as 20 examples per class. Similarly, Large Language Models (LLMs) like GPT-4 can perform effectively with just 1-5 examples per class. However, the performance-cost trade-offs of these methods remain underexplored, a critical concern for budget-limited organizations. Our work addresses this gap by studying the aforementioned approaches over the Banking77 financial intent detection dataset, including the evaluation of cutting-edge LLMs by OpenAI, Cohere, and Anthropic in a comprehensive set of few-shot scenarios. We complete the picture with two additional methods: first, a cost-effective querying method for LLMs based on retrieval-augmented generation (RAG), able to reduce operational costs multiple times compared to classic few-shot approaches, and second, a data augmentation method using GPT-4, able to improve performance in data-limited scenarios. Finally, to inspire future research, we provide a human expert's curated subset of Banking77, along with extensive error analysis. CCS CONCEPTS• Computing methodologies → Natural language processing.

show abstract

Section: Llms For Data Generation In Low-resource Settingsmentioning

confidence: 99%