Query Refinement Prompts for Closed-Book Long-Form Question Answering

Abstract

Large language models (LLMs) have been shown to perform well in answering questions and in producing long-form texts, both in few-shot closed-book settings. While the former can be validated using well-known evaluation metrics, the latter is difficult to evaluate. We resolve the difficulties to evaluate long-form output by doing both tasks at once -to do question answering that requires long-form answers. Such questions tend to be multifaceted, i.e., they may have ambiguities and/or require information from multiple sources. To this end, we define query refinement prompts that encourage LLMs to explicitly express the multifacetedness in questions and generate long-form answers covering multiple facets of the question. Our experiments on two long-form question answering datasets, ASQA and AQuAMuSe, show that using our prompts allows us to outperform fully finetuned models in the closed book setting, as well as achieve results comparable to retrieve-then-generate open-book models.

Evaluation of Medium-large Language Models at Zero-shot Closed Book Generative Question Answering

“…The respective ML tasks are called long-form question answering [15] which was originally designed to involve document retrieval before answering the question. However, LLMs and to some degree also MLMs should be capable of performing it as closed-book QA [16]. This poses the problem that evaluation of results is difficult due to ambiguity of questions (ibid) and other challenges.…”

PreWoMe: Exploiting Presuppositions as Working Memory for Long Form Question Answering

“…Desirable answers to misleading questions require discerning and resolving misleadings. A previous work by Amplayo et al (2022) focuses on ambiguous questions, by introducing query refinement prompts that encourage LMs to consider multiple facets of the question. Another work (Kim et al, 2021) tackles questions containing FP, by extracting and verifying presuppositions.…”