Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study

Yao, Zhiliang; Su, Yu; Sun, Huan; Yih, Wen-tau

doi:10.18653/v1/d19-1547

Cited by 48 publications

(55 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Notably, Thomason et al (2019) used this conversational structure in a robotics setting similar to ours, but focused on learning new percepts, rather than structural abstractions. Yao et al (2019) defined a similar conversational system for Text-to-SQL models that decides when intervention is needed, and generates a clarification question accordingly.…”

Section: Related Workmentioning

confidence: 99%

Learning Adaptive Language Interfaces through Decomposition

Karamcheti¹,

Sadigh²,

Liang³

2020

Proceedings of the First Workshop on Interactive and Executable Semantic Parsing

View full text Add to dashboard Cite

Our goal is to create an interactive natural language interface that efficiently and reliably learns from users to complete tasks in simulated robotics settings. We introduce a neural semantic parsing system that learns new high-level abstractions through decomposition: users interactively teach the system by breaking down high-level utterances describing novel behavior into low-level steps that it can understand. Unfortunately, existing methods either rely on grammars which parse sentences with limited flexibility, or neural sequence-to-sequence models that do not learn efficiently or reliably from individual examples. Our approach bridges this gap, demonstrating the flexibility of modern neural systems, as well as the one-shot reliable generalization of grammar-based methods. Our crowdsourced interactive experiments suggest that over time, users complete complex tasks more efficiently while using our system by leveraging what they just taught. At the same time, getting users to trust the system enough to be incentivized to teach high-level utterances is still an ongoing challenge. We end with a discussion of some of the obstacles we need to overcome to fully realize the potential of the interactive paradigm.

show abstract

Section: Related Workmentioning

confidence: 99%

Learning Adaptive Language Interfaces through Decomposition

Karamcheti¹,

Sadigh²,

Liang³

2020

Proceedings of the First Workshop on Interactive and Executable Semantic Parsing

View full text Add to dashboard Cite

show abstract

“…Learning from weak supervision and user feedback: Another approach to reducing annotation costs is changing full supervision to some form of weak (but potentially noisier) supervision. This has been adopted for various tasks such as machine translation (Saluja, 2012;Petrushkov et al, 2018;Clark et al, 2018;Kreutzer and Riezler, 2019), semantic parsing (Clarke et al, 2010;Liang et al, 2017;Talmor and Berant, 2018), or interactive systems that learn from user interactions (Iyer et al, 2017;Gur et al, 2018;Yao et al, 2019Yao et al, , 2020. For instance, Iyer et al (2017) used users to flag incorrect SQL queries.…”

Section: Related Workmentioning

confidence: 99%

Learning a Cost-Effective Annotation Policy for Question Answering

Kratzwald¹,

Feuerriegel²,

Sun³

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

Self Cite

View full text Add to dashboard Cite

State-of-the-art question answering (QA) relies upon large amounts of training data for which labeling is time consuming and thus expensive. For this reason, customizing QA systems is challenging. As a remedy, we propose a novel framework for annotating QA datasets that entails learning a cost-effective annotation policy and a semi-supervised annotation scheme. The latter reduces the human effort: it leverages the underlying QA system to suggest potential candidate annotations. Human annotators then simply provide binary feedback on these candidates. Our system is designed such that past annotations continuously improve the future performance and thus overall annotation cost. To the best of our knowledge, this is the first paper to address the problem of annotating questions with minimal annotation cost. We compare our framework against traditional manual annotations in an extensive set of experiments. We find that our approach can reduce up to 21.1% of the annotation cost.

show abstract

“…But it is designed for relatively simple scenarios. In this research area, another impressive work involves a modelbased interaction system, which detects uncertain tokens and asks questions relying on inner parser states (Yao et al, 2019). Unlike these studies, however, we design a parser-independent interactive approach that can also perform cross-domain complex SQL queries.…”

Section: Related Workmentioning

confidence: 99%

“What Do You Mean by That?” A Parser-Independent Interactive Approach for Enhancing Text-to-SQL

Chen

Qian

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

In Natural Language Interfaces to Databases systems, the text-to-SQL technique allows users to query databases by using natural language questions. Though significant progress in this area has been made recently, most parsers may fall short when they are deployed in real systems. One main reason stems from the difficulty of fully understanding the users' natural language questions. In this paper, we include human in the loop and present a novel parser-independent interactive approach (PIIA) that interacts with users using multichoice questions and can easily work with arbitrary parsers. Experiments were conducted on two cross-domain datasets, the WikiSQL and the more complex Spider, with five state-ofthe-art parsers. These demonstrated that PIIA is capable of enhancing the text-to-SQL performance with limited interaction turns by using both simulation and human evaluation. User NL question: List the creation year, name and budget of each department. Corrected NL question: List the creation year, name and budget in billions of each department. Modifier Rule: [Noun; C] → [C] (replace the token with the whole column name) 2 User NL question: What are the first names of all the different drivers in alphabetical order? Corrected NL question: What are the forename of all the different drivers in alphabetical order? Modifier Rule: [Noun; C] → [C] (detect the phrase and replace it with the whole column name) 3 User NL question: What is the type of the document named "David CV"? Corrected NL question: What is the document type code of the document whose name is "David CV"? Modifier Rule: [Verb; C] → whose [C] is (directly tell the column name of the verb token) 4 User NL question: Where is the club "Hopkins Student Enterprises" located? Corrected NL question: Where is the club "Hopkins Student Enterprises" 's location? Modifier Rule: [Verb; C] → 's [C] (directly tell the column name of the verb token, when the token is in the end of the sentence) 5 User NL question: Show the enrollment and primary_conference of the oldest college. Corrected NL question: Show the enrollment and primary_conference of the oldest founded college.

show abstract

Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study

Cited by 48 publications

References 34 publications

Learning Adaptive Language Interfaces through Decomposition

Learning Adaptive Language Interfaces through Decomposition

Learning a Cost-Effective Annotation Policy for Question Answering

“What Do You Mean by That?” A Parser-Independent Interactive Approach for Enhancing Text-to-SQL

Contact Info

Product

Resources

About