Improving Compositional Generalization with Latent Structure and Data Augmentation

Abstract

Generic unstructured neural networks have been shown to struggle on out-of-distribution compositional generalization. Compositional data augmentation via example recombination has transferred some prior knowledge about compositionality to such black-box neural models for several semantic parsing tasks, but this often required task-specific engineering or provided limited gains.We present a more powerful data recombination method using a model called Compositional Structure Learner (CSL). CSL is a generative model with a quasi-synchronous context-free grammar backbone, which we induce from the training data. We sample recombined examples from CSL and add them to the fine-tuning data of a pre-trained sequence-tosequence model (T5). This procedure effectively transfers most of CSL's compositional bias to T5 for diagnostic tasks, and results in a model even stronger than a T5-CSL ensemble on two real world compositional generalization tasks. This results in new state-ofthe-art performance for these challenging semantic parsing tasks requiring generalization to both natural language variation and novel compositions of elements. * Equal contribution. † Work done as part of the Google AI Residency program. 1 Also commonly referred to as elements or concepts.

Compositional generalization with a broad-coverage semantic parser

“…On structural generalization in particular, the accuracy of all these models is below 10%, with the exception of Zheng and Lapata (2021), who achieve 39% on PP recursion. By contrast, the compositional model of Liu et al (2021) and the model of Qiu et al (2022), which uses compositional data augmentation, achieve accuracies upwards of 98% on the full generalization set.…”

“…This points to a fundamental tension between broad-coverage semantic parsing on natural text and the ability to generalize compositionally from structurally limited synthetic training sets (see also Shaw et al, 2021). To our knowledge, the only parser that does well on both is the CSL-T5 system of Qiu et al (2022), which fine-tunes T5 using a complex data augmentation (DA) method involving synchronous grammars.…”

Grammar-based Decoding for Improved Compositional Generalization in Semantic Parsing

“…To also address natural language variations in non-synthetic tasks, some recent works exploit structure of the source input and its relation to the target side (Herzig and Berant, 2021;Weißenhorn et al, 2022), and employ sourceside parsing that can be computationally demanding for long sentences, and may have coverage challenge and not available in all languages; while we try to exploit target-side structure only for higher efficiency. Some other works leverage source-side structure for data augmentation to overcome distribution divergence (Yang et al, 2022b;Qiu et al, 2022), which can clearly help but is not the focus of this paper. Grammar-based decoding has shown to help semantic parsing on in-distribution data (Krishnamurthy et al, 2017;Yin and Neubig, 2017).…”

Hierarchical Phrase-based Sequence-to-Sequence Learning

“…More generally our approach extends the recent line of work on neural parameterizations of classic grammars (Jiang et al, 2016;Han et al, 2017Han et al, , 2019Kim et al, 2019;Jin et al, 2019;Zhu et al, 2020;Yang et al, 2021b,a;Zhao and Titov, 2020, inter alia), although unlike in these works we focus on the transduction setting. Data Augmentation Our work is also related to the line of work on utilizing grammatical or alignment structures to guide flexible neural seq2seq models via data augmentation (Jia and Liang, 2016;Fadaee et al, 2017;Andreas, 2020;Akyürek et al, 2021;Shi et al, 2021;Yang et al, 2022;Qiu et al, 2022) or auxiliary supervision (Cohn et al, 2016;Mi et al, 2016;Liu et al, 2016;. In contrast to these works our data augmentation module has stronger inductive biases for hierarchical structure due to explicit use of latent tree-based alignments.…”