Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback

Abstract

We study the task of semantic parse correction with natural language feedback. Given a natural language utterance, most semantic parsing systems pose the problem as one-shot translation where the utterance is mapped to a corresponding logical form. In this paper, we investigate a more interactive scenario where humans can further interact with the system by providing free-form natural language feedback to correct the system when it generates an inaccurate interpretation of an initial utterance. We focus on natural language to SQL systems and construct, SPLASH, a dataset of utterances, incorrect SQL interpretations and the corresponding natural language feedback. We compare various reference models for the correction task and show that incorporating such a rich form of feedback can significantly improve the overall semantic parsing accuracy while retaining the flexibility of natural language interaction. While we estimated human correction accuracy is 81.5%, our best model achieves only 25.1%, which leaves a large gap for improvement in future research.

An Imitation Game for Learning Semantic Parsers from User Interaction

“…Interactive Semantic Parsing. Our work extends the recent idea of leveraging system-user interaction to improve semantic parsing on the fly (Li and Jagadish, 2014;He et al, 2016;Chaurasia and Mooney, 2017;Gur et al, 2018;Yao et al, 2019a,b;Elgohary et al, 2020;Zeng et al, 2020;Semantic Machines et al, 2020). Gur et al (2018) built a neural model to identify and correct error spans in generated queries via dialogues.…”

“…Yao et al (2019b) formalized a model-based intelligent agent MISP, which enables user interaction via a policy probability-based uncertainty estimator, a grammar-based natural language generator, and a multi-choice question-answer interaction design. More recently, Elgohary et al (2020) crowdsourced a dataset for fixing incorrect SQL queries using free-form natural language feedback. Semantic Machines et al (2020) constructed a contextual semantic parsing dataset where agents could trigger conversations to handle exceptions such as ambigu-ous or incomplete user commands.…”

Context Dependent Semantic Parsing: A Survey

“…Feedback/Interactive Semantic Parsing is another line of research in semantic parsing that utilizes context to refine MRs in an iterative manner. Most Feedback Semantic Parsing systems (Iyer et al, 2017;Yao et al, 2019b;Yao et al, 2019a;Elgohary et al, 2020) start with using an CISP parser to parse a given utterance into an initial MR. Then the MR is interpreted in natural language and sent to a user. The user provides feedback, based on which the systems revise the initial parse.…”

“…In contrast, CDSP focuses on modelling the dependencies between the utterances. Elgohary et al (2020) empirically compares CDSP with Feedback Semantic Parsing. They train a CDSP model, EditSQL (Zhang et al, 2019), on two CDSP datasets, SPARC and COSQL, and evaluate it on the test set of a feedback semantic parsing dataset, SPLASH.…”

NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction

“…Unlike other frameworks for interactive semantic parsing that typically expect users to judge the correctness of the execution result or induced logical form, Elgohary et al (2020) introduced a framework for interactive text-to-SQL in which induced SQL queries are fully explained in natural lan-guage to users, who in turn, can correct such parses through natural language feedback (Figure 1). They construct the SPLASH dataset and use it to evaluate baselines for the semantic parse correction with natural language feedback task they introduce.…”