In-Context Learning for Few-Shot Dialogue State Tracking

Hu, Yushi; Lee, Chia-Hsuan; Xie, Tianbao; Chen, Yu; Smith, Noah A.; Ostendorf, Mari

doi:10.18653/v1/2022.findings-emnlp.193

Cited by 20 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Pretrained with the internet corpora, LLMs are already familiar with the syntax of formal query languages such as SQL (Hu et al, 2022;Poesia et al, 2022;Arora et al, 2023). When given simple SQL schemas, they can perform zero-shot semantic parsing of simple natural language queries into formal queries.…”

Section: Few-shot Seq2seq Semantic Parsingmentioning

confidence: 99%

“…Niu et al (2023) use a few-shot prompted Codex model to break down the natural language input to make the task easier for a smaller semantic parser. LLMs have also been applied to semantic parsing on relational databases (Hu et al, 2022;Poesia et al, 2022;Arora et al, 2023). The schemas used in these projects are very small when compared to Wikidata.…”

Section: Kbqa Benchmarksmentioning

confidence: 99%

See 1 more Smart Citation

Fine-tuned LLMs Know More, Hallucinate Less with Few-Shot Sequence-to-Sequence Semantic Parsing over Wikidata

Xu,

Liu,

Culhane

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

While large language models (LLMs) can answer many questions correctly, they can also hallucinate and give wrong answers. Wikidata, with its over 12 billion facts, can be used to ground LLMs to improve their factuality. This paper presents WikiWebQuestions, a highquality question answering benchmark for Wikidata. Ported over from WebQuestions for Freebase, it consists of real-world data with SPARQL annotation. This paper presents a few-shot sequence-tosequence semantic parser for Wikidata. We modify SPARQL to use the unique domain and property names instead of their IDs. We train the parser to use either the results from an entity linker or mentions in the query. We fine-tune LLaMA by adding the few-shot training data to that used to fine-tune Alpaca.Our experimental results demonstrate the effectiveness of this methodology, establishing a strong baseline of 76% and 65% answer accuracy in the dev and test sets of WikiWeb-Questions, respectively. By pairing our semantic parser with GPT-3, we combine verifiable results with qualified GPT-3 guesses to provide useful answers to 96% of the questions in dev. We also show that our method outperforms the state-of-the-art for the QALD-7 Wikidata dataset by 3.6% in F1 score. 1

show abstract

Section: Few-shot Seq2seq Semantic Parsingmentioning

confidence: 99%

Section: Kbqa Benchmarksmentioning

confidence: 99%

Fine-tuned LLMs Know More, Hallucinate Less with Few-Shot Sequence-to-Sequence Semantic Parsing over Wikidata

Xu,

Liu,

Culhane

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…LLM DST. IC-DST (Hu et al, 2022) is an incontext learning (ICL) framework that enables fewshot DST with LLMs. The prediction is the change in each turn pair instead of the accumulated dialogue states.…”

Section: Dialogue State Trackingmentioning

confidence: 99%

“…For DST, ; Hudeček and Dušek (2023) prompts LLM with human-authored task descriptions or incontext exemplars. Hu et al (2022) improves the in-context learning for DST performance by incorporating a retriever to fetch useful exemplars. King and Flanigan (2023) further increases the diversity of in-context exemplars and improves the decoding mechanism.…”

Section: Introductionmentioning

confidence: 99%

Dialogue State Tracking with a Language Model using Schema-Driven Prompting

Lee¹,

Cheng²,

Ostendorf³

2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

Self Cite

View full text Add to dashboard Cite

Large language models (LLMs) have revolutionized the landscape of Natural Language Processing systems, but are computationally expensive. To reduce the cost without sacrificing performance, previous studies have explored various approaches to harness the potential of Small Language Models (SLMs) as cost-effective alternatives to their larger counterparts. Driven by findings that SLMs and LLMs exhibit complementary strengths in a structured knowledge extraction task, this work presents a novel SLM/LLM routing framework designed to improve computational efficiency and enhance task performance. First, exemplar pools are created to represent the types of contexts where each LM provides a more reliable answer, leveraging a sentence embedding fine-tuned so that context similarity is close to dialogue state similarity. Then, during inference, the k-nearest exemplars to the testing instance are retrieved, and the instance is routed according to majority vote. In dialogue state tracking tasks, the proposed routing framework enhances performance substantially compared to relying solely on LLMs, while reducing the computational costs by over 50%.

show abstract

“…

…”

mentioning

confidence: 99%

Measuring and Mitigating Constraint Violations of In-Context Learning for Utterance-to-API Semantic Parsing

Wang,

Jean,

Sengupta

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

In executable task-oriented semantic parsing, the system aims to translate users' utterances in natural language to machine-interpretable programs (API calls) that can be executed according to pre-defined API specifications. With the popularity of Large Language Models (LLMs), in-context learning offers a strong baseline for such scenarios, especially in data-limited regimes (Hu et al., 2022;. However, LLMs are known to hallucinate and therefore pose a formidable challenge in constraining generated content (Parikh et al., 2020). Thus, it remains uncertain if LLMs can effectively perform task-oriented utterance-to-API generation where respecting API's structural and task-specific constraints is crucial. In this work, we seek to measure, analyze and mitigate such constraints violations. First, we identify the categories of various constraints in obtaining API-semantics from task-oriented utterances, and define fine-grained metrics that complement traditional ones. Second, we leverage these metrics to conduct a detailed error analysis of constraints violations seen in state-of-theart LLMs, which motivates us to investigate two popular mitigation strategies-Semantic-Retrieval of Demonstrations (SRD) and APIaware Constrained Decoding (API-CD). Our experiments show that these strategies are effective at reducing constraints violations and improving the quality of the generated API calls, but require careful consideration given their implementation complexity and latency.

show abstract

In-Context Learning for Few-Shot Dialogue State Tracking

Cited by 20 publications

References 0 publications

Fine-tuned LLMs Know More, Hallucinate Less with Few-Shot Sequence-to-Sequence Semantic Parsing over Wikidata

Fine-tuned LLMs Know More, Hallucinate Less with Few-Shot Sequence-to-Sequence Semantic Parsing over Wikidata

Dialogue State Tracking with a Language Model using Schema-Driven Prompting

Measuring and Mitigating Constraint Violations of In-Context Learning for Utterance-to-API Semantic Parsing

Contact Info

Product

Resources

About