Compositional generalization with a broad-coverage semantic parser

Weißenhorn, Pia; Donatelli, Lucia; Koller, Alexander

doi:10.18653/v1/2022.starsem-1.4

Cited by 9 publications

(15 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For small-scale synthetic data, many specialized model architectures proved to be effective on SCAN like tasks Russin et al, 2019;Gordon et al, 2020;Lake, 2019;Liu et al, 2020a;Nye et al, 2020;Chen et al, 2020). To also address natural language variations in non-synthetic tasks, some recent works exploit structure of the source input and its relation to the target side (Herzig and Berant, 2021;Weißenhorn et al, 2022), and employ sourceside parsing that can be computationally demanding for long sentences, and may have coverage challenge and not available in all languages; while we try to exploit target-side structure only for higher efficiency. Some other works leverage source-side structure for data augmentation to overcome distribution divergence (Yang et al, 2022b;Qiu et al, 2022), which can clearly help but is not the focus of this paper.…”

Section: Modelingmentioning

confidence: 99%

Grammar-based Decoding for Improved Compositional Generalization in Semantic Parsing

Zheng¹,

Chow²,

Shen³

et al. 2023

Findings of the Association for Computational Linguistics: ACL 2023

View full text Add to dashboard Cite

Sequence-to-sequence (seq2seq) models have achieved great success in semantic parsing tasks, but they tend to struggle on out-of-distribution (OOD) data. Despite recent progress, robust semantic parsing on large-scale tasks that combine challenges from both compositional generalization and natural language variations remains an unsolved issue. To encourage research in this area, this work introduces CUDON, a large-scale dialogue dataset in the Chinese language, specifically created to evaluate the compositional generalization of semantic parsing. The dataset contains about ten thousand multi-turn complex queries, and provides multiple splits with different degrees of train-test distribution divergence. We have investigated improving compositional generalization through grammar-based decoding on this dataset. With specially designed grammars that leverage program schema, we are able to significantly improve the accuracy of seq2seq semantic parsers on OOD splits: a LSTM-based parser using a Context-free Grammar (CFG) achieves over 25% higher accuracy than a standard seq2seq baseline; a parser using Tree-Substitution Grammar (TSG) improves parsing speed by five to seven times over the CFG parser with only a small accuracy loss. The grammar-based LSTM parsers also outperforms BART-and T5-based seq2seq parsers on the OOD splits, despite having less than one tenth of the parameters and no pretraining. We also validated our approach on the SMCalflow-CS dataset, specifically on the zero-shot learning task.

show abstract

Section: Modelingmentioning

confidence: 99%

Grammar-based Decoding for Improved Compositional Generalization in Semantic Parsing

Zheng¹,

Chow²,

Shen³

et al. 2023

Findings of the Association for Computational Linguistics: ACL 2023

View full text Add to dashboard Cite

show abstract

“…A bunch of synthetically-generated datasets have been created for assessing compositional generalization (Lake and Baroni, 2017;Bastings et al, 2018;Keysers et al, 2020;Kim and Linzen, 2020), and plain sequence-to-sequence(seq2seq) models exhibit significant out-of-distribution (OOD) compositional generalization performance loss comparing to in-distribution (ID) setting. While effective methods have been proposed to overcome the difficulty in OOD compositional generalization Nye et al, 2020;Weißenhorn et al, 2022), most of them mainly focus on semantic parsing, where some important abilities like summarization Dataset # of samples generalization forms GEOQUERY (Shaw et al, 2021) 880 3 SPIDER-SSP (Shaw et al, 2021) 4,376 3 SMCALFLOW-CS (Yin et al, 2021) 28,054 2 COUNTERFACTUAL (Liu et al, 2022) 2,500 1 DINER (ours) 223,581 4…”

Section: Instructionsmentioning

confidence: 99%

DiNeR: A Large Realistic Dataset for Evaluating Compositional Generalization

Hu,

Liu,

Feng

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Most of the existing compositional generalization datasets are synthetically-generated, resulting in a lack of natural language variation. While there have been recent attempts to introduce non-synthetic datasets for compositional generalization, they suffer from either limited data scale or a lack of diversity in the forms of combinations. To better investigate compositional generalization with more linguistic phenomena and compositional diversity, we propose the DIsh NamE Recognition (DINER) task and create a large realistic Chinese dataset. Given a recipe instruction, models are required to recognize the dish name composed of diverse combinations of food, actions, and flavors. Our dataset consists of 3,811 dishes and 228,114 recipes, and involves plenty of linguistic phenomena such as anaphora, omission and ambiguity. We provide two strong baselines based on T5 (Raffel et al., 2020) and large language models (LLMs). This work contributes a challenging task, baseline methods to tackle the task, and insights into compositional generalization in the context of dish name recognition.

show abstract

“…Moreover, large pre-trained language models have been shown not to improve compositional generalization (Oren et al, 2020;Qiu et al, 2022b). This has prompted the community to realize that parsers should be designed intentionally with compositionality in mind (Lake, 2019;Gordon et al, 2020;Weißenhorn et al, 2022).…”

Section: Related Workmentioning

confidence: 99%

Translate First Reorder Later: Leveraging Monotonicity in Semantic Parsing

Cazzaro,

Locatelli,

Quattoni

et al. 2023

Findings of the Association for Computational Linguistics: EACL 2023

View full text Add to dashboard Cite

Prior work in semantic parsing has shown that conventional seq2seq models fail at compositional generalization tasks. This limitation led to a resurgence of methods that model alignments between sentences and their corresponding meaning representations, either implicitly through latent variables or explicitly by taking advantage of alignment annotations. We take the second direction and propose TPOL, a two-step approach that first translates input sentences monotonically and then reorders them to obtain the correct output. This is achieved with a modular framework comprising a Translator and a Reorderer component. We test our approach on two popular semantic parsing datasets. Our experiments show that by means of the monotonic translations, TPOL can learn reliable lexico-logical patterns from aligned data, significantly improving compositional generalization both over conventional seq2seq models, as well as over other approaches that exploit gold alignments. Our code is publicly available at https://github. com/interact-erc/TPol.git

show abstract

Compositional generalization with a broad-coverage semantic parser

Cited by 9 publications

References 18 publications

Grammar-based Decoding for Improved Compositional Generalization in Semantic Parsing

Grammar-based Decoding for Improved Compositional Generalization in Semantic Parsing

DiNeR: A Large Realistic Dataset for Evaluating Compositional Generalization

Translate First Reorder Later: Leveraging Monotonicity in Semantic Parsing

Contact Info

Product

Resources

About