Selective In-Context Data Augmentation for Intent Detection using Pointwise V-Information

Lin, Yen-Ting; Papangelis, Alexandros; Kim, Seokhwan; Lee, Sungjin; Hazarika, Devamanyu; Namazifar, Mahdi; Jin, Di; Liu, Yang; Hakkani-Tur, Dilek

doi:10.18653/v1/2023.eacl-main.107

Cited by 6 publications

(4 citation statements)

References 34 publications

(49 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Baseline For intent classification, we compare with various few-shot fine-tuned baselines: RoBERTa (Liu et al, 2019), ICDA (Lin et al, 2023), DNNC (Zhang et al, 2020a), and CPFT (Zhang et al, 2021). While for zeroshot baseline, we employ MNLI (Williams et al, 2018) fine-tuned BART large models (Lewis et al, 2020) by framing intent classification as an NLI task.…”

Section: Experiments Settingsmentioning

confidence: 99%

InstructTODS: Large Language Models for End-to-End Task-Oriented Dialogue Systems

Chung,

Cahyawijaya,

Wilie

et al. 2023

Proceedings of the Second Workshop on Natural Language Interfaces

View full text Add to dashboard Cite

Large language models (LLMs) have been used for diverse tasks in natural language processing (NLP), yet remain under-explored for task-oriented dialogue systems (TODS), especially for end-to-end TODS. We present In-structTODS, a novel off-the-shelf framework for zero-shot end-to-end task-oriented dialogue systems that can adapt to diverse domains without fine-tuning. By leveraging LLMs, Instruct-TODS generates a proxy belief state that seamlessly translates user intentions into dynamic queries for efficient interaction with any KB. Our extensive experiments demonstrate that In-structTODS achieves comparable performance to fully fine-tuned TODS in guiding dialogues to successful completion without prior knowledge or task-specific data. Furthermore, a rigorous human evaluation of end-to-end TODS shows that InstructTODS produces dialogue responses that notably outperform both the gold responses and the state-of-the-art TODS in terms of helpfulness, informativeness, and humanness. Moreover, the effectiveness of LLMs in TODS is further supported by our comprehensive evaluations on TODS subtasks: dialogue state tracking, intent classification, and response generation. Code and implementations could be found here 1 . Anonymous. 2023. Nusawrites: Constructing highquality corpora for underrepresented and extremely low-resource languages. Anonymous preprint under review.

show abstract

Section: Experiments Settingsmentioning

confidence: 99%

InstructTODS: Large Language Models for End-to-End Task-Oriented Dialogue Systems

Chung,

Cahyawijaya,

Wilie

et al. 2023

Proceedings of the Second Workshop on Natural Language Interfaces

View full text Add to dashboard Cite

show abstract

“…It is used as a quality estimate of Universal Dependencies treebanks in Kulmizev and Nivre (2023). PVI is used to select synthetic data as an augmentation to an intent detection classifier, which achieves state-of-the-art performance (Lin et al, 2023). Chen et al (2022a) and Prasad et al (2023) incorporate PVI into an informativeness metric to evaluate rationales, and find that it captures the expected flow of information in high-quality reasoning chains.…”

Section: Hardnessmentioning

confidence: 99%

“…PVI measures the amount of usable information in an input for a given model, which reflects the ease with which a model can predict a certain label given an input. Though it is a recently proposed method, the effectiveness of PVI has been demonstrated in various NLP tasks (Chen et al, 2022a;Kulmizev and Nivre, 2023;Lin et al, 2023;Prasad et al, 2023).…”

Section: Introductionmentioning

confidence: 99%

Measuring Pointwise 𝒱-Usable Information In-Context-ly-Usable Information In-Context-ly

Lu,

Chen,

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

In-context learning (ICL) is a new learning paradigm that has gained popularity along with the development of large language models. In this work, we adapt a recently proposed hardness metric, pointwise V-usable information (PVI), to an in-context version (in-context PVI). Compared to the original PVI, in-context PVI is more efficient in that it requires only a few exemplars and does not require fine-tuning. We conducted a comprehensive empirical analysis to evaluate the reliability of in-context PVI.Our findings indicate that in-context PVI estimates exhibit similar characteristics to the original PVI. Specific to the in-context setting, we show that in-context PVI estimates remain consistent across different exemplar selections and numbers of shots. The variance of in-context PVI estimates across different exemplar selections is insignificant, which suggests that incontext estimates PVI are stable. Furthermore, we demonstrate how in-context PVI can be employed to identify challenging instances. Our work highlights the potential of in-context PVI and provides new insights into the capabilities of ICL. 1

show abstract

“…Task As a fundamental element in task-oriented dialog systems, intent detection is normally conducted in the NLU component for identifying a user's intent given an utterance (Ham et al, 2020). Recently, accurately identifying intents in the fewshot setting has attracted much attention due to data scarcity issues resulted from the cost of data collection as well as privacy and ethical concerns (Lin et al, 2023b). Following the few-shot intent detection benchmark , we focus on the challenging 5-shot and 10-shot settings.…”

Section: Few-shot Intent Classificationmentioning

confidence: 99%

Dense Retrieval as Indirect Supervision for Large-space Decision Making

Xu,

Wang,

Dong

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Many discriminative natural language understanding (NLU) tasks have large label spaces. Learning such a process of large-space decision making is particularly challenging due to the lack of training instances per label and the difficulty of selection among many fine-grained labels. Inspired by dense retrieval methods for passage finding in open-domain QA, we propose a reformulation of large-space discriminative NLU tasks as a learning-to-retrieve task, leading to a novel solution named Dense Decision Retrieval (DDR ). Instead of predicting fine-grained decisions as logits, DDR adopts a dual-encoder architecture that learns to predict by retrieving from a decision thesaurus. This approach not only leverages rich indirect supervision signals from easy-to-consume learning resources for dense retrieval, it also leads to enhanced prediction generalizability with a semantically meaningful representation of the large decision space. When evaluated on tasks with decision spaces ranging from hundreds to hundred-thousand scales, DDR outperforms strong baselines greatly by 27.54% in P @1 on two extreme multi-label classification tasks, 1.17% in F1 score ultra-fine entity typing, and 1.26% in accuracy on three few-shot intent classification tasks on average. 1

show abstract

Selective In-Context Data Augmentation for Intent Detection using Pointwise V-Information

Cited by 6 publications

References 34 publications

InstructTODS: Large Language Models for End-to-End Task-Oriented Dialogue Systems

InstructTODS: Large Language Models for End-to-End Task-Oriented Dialogue Systems

Measuring Pointwise 𝒱-Usable Information In-Context-ly-Usable Information In-Context-ly

Dense Retrieval as Indirect Supervision for Large-space Decision Making

Contact Info

Product

Resources

About