Exploring Unexplored Generalization Challenges for Cross-Database Semantic Parsing

Abstract

We study the task of cross-database semantic parsing (XSP), where a system that maps natural language utterances to executable SQL queries is evaluated on databases unseen during training. Recently, several datasets, including Spider, were proposed to support development of XSP systems. We propose a challenging evaluation setup for cross-database semantic parsing, focusing on variation across database schemas and in-domain language use. We re-purpose eight semantic parsing datasets that have been well-studied in the setting where in-domain training data is available, and instead use them as additional evaluation data for XSP systems instead. We build a system that performs well on Spider, and find that it struggles to generalize to our re-purposed set. Our setup uncovers several generalization challenges for cross-database semantic parsing, demonstrating the need to use and develop diverse training and evaluation datasets. * Work done during an internship at Google. Advising (Finegan-Dollak et al., 2018) NL: For EECS 478, how many credits is it? SQL: select distinct credits from course where department ='EECS' and number = 478; GeoQuery (Zelle and Mooney, 1996) NL: How many people live in mississippi? SQL: select population from state where state name = 'mississippi'; ATIS (Hemphill et al., 1990; Dahl et al., 1994) NL: Flights from Phoenix to Milwaukee SQL: select distinct T1.flight id from airport service as T2, airport service as T3, city as T4, city as T5, flight as T1 where T4.city code = T2.city code and T4.city name = 'Phoenix' and T5.city code = T3.city code and T5.city name = 'Milwaukee' and T1.from airport = T2.airport code and T1.to airport = T3.airport code; Spider (Yu et al., 2018) NL: List the emails of the professionals who live in the state of Hawaii or the state of Wisconsin.

Structure-Grounded Pretraining for Text-to-SQL

“…(2) On more realistic evaluation settings, including Spider-Realistic and the Suhr et al (2020) datasets, our method outperforms all baselines. This demonstrates the superiority of our pretraining framework in solving the text-table alignment challenge, and its usefulness in practice.…”

“…As pointed out by Suhr et al (2020), existing text-to-SQL benchmarks like Spider (Yu et al, 2018b) render the text-table alignment challenge easier than expected by explicitly mentioning exact column names in the NL utterances. Contrast this to more realistic settings where users may refer to the columns using a variety of expressions.…”

“…Contrast this to more realistic settings where users may refer to the columns using a variety of expressions. Suhr et al (2020) propose a new cross-database setting that uses Spider for training and includes eight other single-domain text-to-SQL datasets for evaluation. In addition to adopting their setting, we create a new evaluation set called Spider-Realistic from the original Spider dev set, by removing explicit mentions of column names from an utterance.…”

Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsing

“…More recently, large-scale datasets consisting of hundreds of DBs and the corresponding question-SQL pairs have been released Zhong et al, 2017;Yu et al, 2019b,a) to encourage the development of semantic parsers that can work well across different DBs (Guo et al, 2019;Bogin et al, 2019b;Wang et al, 2019;Suhr et al, 2020;Choi et al, 2020). The setup is challenging as it requires the model to interpret a question conditioned on a relational DB unseen during training and accurately express the question intent via SQL logic.…”

A Tale of Two Linkings: Dynamically Gating between Schema Linking and Structural Linking for Text-to-SQL Parsing

“…However, it is still difficult for current state-of-the-art models to fill in the skeletons with semantically correct entities, especially when they are required to generalize to unseen DB schemas (Yu et al, 2018;Suhr et al, 2020). To predict the correct entity, the model should have a database (DB) schema grounded understanding of the NL question, which means that the model should be able to jointly learn the semantics in the NL question and the structured knowledge in a given database.…”