DialSQL: Dialogue Based Structured Query Generation

Abstract

The recent advance in deep learning and semantic parsing has significantly improved the translation accuracy of natural language questions to structured queries. However, further improvement of the existing approaches turns out to be quite challenging. Rather than solely relying on algorithmic innovations, in this work, we introduce DialSQL, a dialoguebased structured query generation framework that leverages human intelligence to boost the performance of existing algorithms via user interaction. DialSQL is capable of identifying potential errors in a generated SQL query and asking users for validation via simple multi-choice questions. User feedback is then leveraged to revise the query. We design a generic simulator to bootstrap synthetic training dialogues and evaluate the performance of DialSQL on the WikiSQL dataset. Using SQLNet as a black box query generation tool, DialSQL improves its performance from 61.3% to 69.0% using only 2.4 validation questions per dialogue.

Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study

“…Most human evaluation studies for (interactive) semantic parsers so far (Chaurasia and Mooney, 2017;Gur et al, 2018;Yao et al, 2019) use pre-existing test questions (e.g., from datasets like WikiSQL). However, this introduces an undesired discrepancy, that is, human evaluators may not necessarily be able to understand the true intent of the given questions in an faithful way, especially when the question is ambiguous, vague, or containing unfamiliar entities.…”

“…Interactive Semantic Parsing. To enhance parsing accuracy and user confidence in practical applications, interactive semantic parsing has emerged as a promising solution (Li and Jagadish, 2014;He et al, 2016;Chaurasia and Mooney, 2017;Gur et al, 2018;Yao et al, 2019). Despite their effectiveness, existing solutions are somewhat ad-hoc and bound to a specific formal language and dataset.…”

“…Despite their effectiveness, existing solutions are somewhat ad-hoc and bound to a specific formal language and dataset. For example, DialSQL (Gur et al, 2018) is curated for Wik-iSQL, where SQL queries all follow the same and given grammar sketch. Similarly, (Yao et al, 2019) relies on a pre-defined two-level hierarchy among components in an If-Then program and cannot generalize to formal languages with a deeper structure.…”

“…We first compare MISP-SQL with the two base semantic parsers without interactions in Table 2. For SQLNet, we also compare our system with the reported performance of DialSQL (Gur et al, 2018, Table 4). However, since DialSQL is not open-sourced and it is not easy to reproduce it, we are unable to adapt it to SQLova for more comparisons.…”

“…In response to these challenges, interactive semantic parsing has been proposed recently as a practical solution, which includes human users in the loop to resolve utterance ambiguity, boost system accuracy, and improve user confidence via human-machine collaboration (Li and Jagadish, 2014;He et al, 2016;Chaurasia and Mooney, 2017;Gur et al, 2018;Yao et al, 2019). For example, Gur et al (2018) built the DialSQL system to detect errors in a generated SQL query and request user selection on alternative options via dialogues. Similarly, Chaurasia and Mooney (2017) and Yao et al (2019) enabled semantic parsers to ask users clarification questions while generating an If-Then program.…”