CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models

Kuhn, Lorenz; Gal, Yarin; Farquhar, Sebastian

doi:10.48550/arxiv.2212.07769

Cited by 3 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, this concept is closely related to question generation [5] and clarification question generation [3,6]. The concept of clarification questions was formally introduced in [7], and since then, research into generating these questions has spanned a wide range of scenarios, including open-domain systems (AmbigQA) [8], knowledge bases (CLAQUA) [6], closed-book systems (CLAM) [9], information-seeking (ISEEQ) [10], task-oriented dialog systems (CLARIT) [3], and conversational search [11]. Rahmani et al [12] surveyed the various methodologies, datasets, and different evaluation strategies used for clarification questions.…”

Section: Related Workmentioning

confidence: 99%

Dynamic Contexts for Generating Suggestion Questions in RAG Based Conversational Systems

Tayal,

Tyagi

2024

Companion Proceedings of the ACM Web Conference 2024

View full text Add to dashboard Cite

When interacting with Retrieval-Augmented Generation (RAG)based conversational agents, the users must carefully craft their queries to be understood correctly. Yet, understanding the system's capabilities can be challenging for the users, leading to ambiguous questions that necessitate further clarification. This work aims to bridge the gap by developing a suggestion question generator. To generate suggestion questions, our approach involves utilizing dynamic context, which includes both dynamic few-shot examples and dynamically retrieved contexts. Through experiments, we show that the dynamic contexts approach can generate better suggestion questions as compared to other prompting approaches. CCS CONCEPTS• Computing methodologies → Natural language generation.

show abstract

Section: Related Workmentioning

confidence: 99%

Dynamic Contexts for Generating Suggestion Questions in RAG Based Conversational Systems

Tayal,

Tyagi

2024

Companion Proceedings of the ACM Web Conference 2024

View full text Add to dashboard Cite

show abstract

“…Amplayo et al ( 2023) suggest optimal prompts specifically engineered for the task. Kuhn et al (2022) prompt LLMs to clarify ambiguous questions selectively. However, the studies do not utilize external information to ensure the factual correctness of the disambiguations, thereby potentially increasing the risk of hallucinations from LLMs.…”

Section: Related Workmentioning

confidence: 99%

Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models

Kim,

Jeon

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Questions in open-domain question answering are often ambiguous, allowing multiple interpretations. One approach to handling them is to identify all possible interpretations of the ambiguous question (AQ) and to generate a long-form answer addressing them all, as suggested by Stelmakh et al. (2022). While it provides a comprehensive response without bothering the user for clarification, considering multiple dimensions of ambiguity and gathering corresponding knowledge remains a challenge. To cope with the challenge, we propose a novel framework, TREE OF CLARIFICATIONS (TOC): It recursively constructs a tree of disambiguations for the AQ-via few-shot prompting leveraging external knowledge-and uses it to generate a long-form answer. TOC outperforms existing baselines on ASQA in a fewshot setup across all metrics, while surpassing fully-supervised baselines trained on the whole training set in terms of Disambig-F1 and Disambig-ROUGE. Code is available at github.com/gankim/tree-of-clarifications.

show abstract

“…Kuhn et al (2023) and Lin et al (2023) use similar sampling-based methods over free-form question answering, using slightly different formulations of confidence scores, but they do not investigate ambiguous questions. Kuhn et al (2022) examine synthetically-created ambiguous questions, but focus on multi-turn interactions.…”

Section: Calibration and Selective Qamentioning

confidence: 99%

Selectively Answering Ambiguous Questions

Cole,

Zhang,

Gillick

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Trustworthy language models should abstain from answering questions when they do not know the answer. However, the answer to a question can be unknown for a variety of reasons. Prior research has focused on the case in which the question is clear and the answer is unambiguous but possibly unknown. But the answer to a question can also be unclear due to uncertainty of the questioner's intent or context. We investigate question answering from this perspective, focusing on answering a subset of questions with a high degree of accuracy, from a set of questions in which many are inherently ambiguous. In this setting, we find that the most reliable approach to decide when to abstain involves quantifying repetition within sampled model outputs, rather than the model's likelihood or self-verification as used in prior work. We find this to be the case across different types of uncertainty and model scales, and with or without instruction tuning. Our results suggest that sampling-based confidence scores help calibrate answers to relatively unambiguous questions, with more dramatic improvements on ambiguous questions.

show abstract

CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models

Cited by 3 publications

References 0 publications

Dynamic Contexts for Generating Suggestion Questions in RAG Based Conversational Systems

Dynamic Contexts for Generating Suggestion Questions in RAG Based Conversational Systems

Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models

Selectively Answering Ambiguous Questions

Contact Info

Product

Resources

About