An Imitation Game for Learning Semantic Parsers from User Interaction

Yao, Zhiliang; Tang, Yiqi; Yih, Wen-tau; Sun, Huan; Su, Yu

doi:10.18653/v1/2020.emnlp-main.559

Cited by 12 publications

(9 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our question decomposition prompting serves as one of the first attempts to mitigate the error propagation issue in LLMs' multi-step reasoning, and we highlight this problem as a meaningful future direction. For example, we can further reduce errors in intermediate reasoning steps by incorporating our method into an interactive semantic parsing framework (Yao et al, 2019(Yao et al, , 2020Li et al, 2020;Zeng et al, 2020;Chen et al, 2023a,b). Since the decomposed sub-questions are in natural language, this interactive approach enables database users to easily spot the errors in each sub-question.…”

Section: Discussionmentioning

confidence: 99%

Exploring Chain of Thought Style Prompting for Text-to-SQL

Tai,

Chen,

Zhang

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

In-context learning with large language models (LLMs) has recently caught increasing attention due to its superior few-shot performance on various tasks. However, its performance on text-to-SQL parsing still has much room for improvement. In this paper, we hypothesize that a crucial aspect of LLMs to improve for text-to-SQL parsing is their multi-step reasoning ability. Thus, we systematically study how to enhance LLMs' reasoning ability through chain of thought (CoT) style prompting, including the original chain-of-thought prompting (Wei et al., 2022b) and least-to-most prompting (Zhou et al., 2023). Our experiments demonstrate that iterative prompting as in Zhou et al. ( 2023) may be unnecessary for text-to-SQL parsing, and using detailed reasoning steps tends to have more error propagation issues. Based on these findings, we propose a new CoT-style prompting method for text-to-SQL parsing. It brings 5.2 and 6.5 point absolute gains on the Spider development set and the Spider Realistic set, respectively, compared to the standard prompting method without reasoning steps; 2.4 and 1.5 point absolute gains, compared to the least-to-most prompting method 1 .

show abstract

Section: Discussionmentioning

confidence: 99%

Exploring Chain of Thought Style Prompting for Text-to-SQL

Tai,

Chen,

Zhang

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…RAT-SQL w. BERT LARGE 62.1 ± 1.3 (58.1) 2.3 ± 0.2 47.3 ± 3.7 37.0 ± 18.9 15.6 ± 2.0 21.8 ± 1.6 16.0 ± 3.1 3.4 ± 1.4 6.4 ± 2.3 w. GRAPPA -(59.3) w. STRUG (Human Assisted) 65.7 ± 0.7 (62.2) 5.5 ± 1.1 59.5 ± 3.2 40.7 ± 13.9 18.7 ± 2.1 26.8 ± 2.9 21.6 ± 2.3 6.3 ± 1.8 6.9 ± 0.6 w. STRUG (Automatic) 65.3 ± 0.7 (62.2) 2.8 ± 0.7 57.5 ± 0.2 44.4 ± 32.7 20.2 ± 1.6 30.2 ± 5.8 18.5 ± 1.5 6.1 ± 0.5 5.2 ± 0.5 (Hwang et al, 2019;Yavuz et al, 2018), mainly because of the simplicity of the SQL queries and large amount of data available for training, previous works have also used this dataset to demonstrate the model's generalization ability with limited training data Yao et al, 2020).…”

Section: Content Usedmentioning

confidence: 99%

Structure-Grounded Pretraining for Text-to-SQL

Deng¹,

Awadallah²,

Meek³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

Learning to capture text-table alignment is essential for tasks like text-to-SQL. A model needs to correctly recognize natural language references to columns and values and to ground them in the given database schema. In this paper, we present a novel weakly supervised Structure-Grounded pretraining framework (STRUG) for text-to-SQL that can effectively learn to capture text-table alignment based on a parallel text-table corpus. We identify a set of novel pretraining tasks: column grounding, value grounding and columnvalue mapping, and leverage them to pretrain a text-table encoder. Additionally, to evaluate different methods under more realistic text-table alignment settings, we create a new evaluation set Spider-Realistic based on Spider dev set with explicit mentions of column names removed, and adopt eight existing textto-SQL datasets for cross-database evaluation. STRUG brings significant improvement over BERT LARGE in all settings. Compared with existing pretraining methods such as GRAPPA, STRUG achieves similar performance on Spider, and outperforms all baselines on more realistic sets. All the code and data used in this work is public available at https://aka.ms/ strug.

show abstract

“…Minecraft as an Environment for Grounded Language Understanding substantiated the advantages of building an open interactive assistant in the sandbox construction game of Minecraft instead of a "real world" assistant, which is inherently complex and inherently costly to develop and maintain. The Minecraft world's constraints (e.g., coarse 3-d voxel grid and simple physics) and the regularities in the head of the distribution of in-game tasks allow numerous scenarios for grounded NLU research [Yao et al, 2020, Srinet et al, 2020. Minecraft is an appealing competition domain due to its popularity as a video game, of all games ever released, it has the second-most total copies sold.…”

Section: Competition Typementioning

confidence: 99%

NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environment

Kiseleva,

Li,

Aliannejadi

et al. 2021

Preprint

View full text Add to dashboard Cite

Human intelligence has the remarkable ability to quickly adapt to new tasks and environments. Starting from a very young age, humans acquire new skills and learn how to solve new tasks either by imitating the behavior of others or by following provided natural language instructions. To facilitate research in this direction, we propose IGLU: Interactive Grounded Language Understanding in a Collaborative Environment.The primary goal of the competition is to approach the problem of how to build interactive agents that learn to solve a task while provided with grounded natural language instructions in a collaborative environment. Understanding the complexity of the challenge, we split it into sub-tasks to make it feasible for participants.This research challenge is naturally related, but not limited, to two fields of study that are highly relevant to the NeurIPS community: Natural Language Understanding and Generation (NLU/G) and Reinforcement Learning (RL). Therefore, the suggested challenge can bring two communities together to approach one of the important challenges in AI. Another important aspect of the challenge is the dedication to perform a human-in-the-loop evaluation as a final evaluation for the agents developed by contestants.

show abstract

An Imitation Game for Learning Semantic Parsers from User Interaction

Cited by 12 publications

References 45 publications

Exploring Chain of Thought Style Prompting for Text-to-SQL

Exploring Chain of Thought Style Prompting for Text-to-SQL

Structure-Grounded Pretraining for Text-to-SQL

NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environment

Contact Info

Product

Resources

About