Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN

Abstract

Despite their failure to solve the compositional SCAN dataset, seq2seq architectures still achieve astonishing success on more practical tasks. This observation pushes us to question the usefulness of SCAN-style compositional generalization in realistic NLP tasks. In this work, we study the benefit that such compositionality brings about to several machine translation tasks. We present several focused modifications of Transformer that greatly improve generalization capabilities on SCAN and select one that remains on par with a vanilla Transformer on a standard machine translation (MT) task. Next, we study its performance in low-resource settings and on a newly introduced distribution-shifted English-French translation task.Overall, we find that improvements of a SCAN-capable model do not directly transfer to the resource-rich MT setup. In contrast, in the low-resource setup, general modifications lead to an improvement of up to 13.1% BLEU score w.r.t. a vanilla Transformer. Similarly, an improvement of 14% in an accuracy-based metric is achieved in the introduced compositional English-French translation task. This provides experimental evidence that the compositional generalization assessed in SCAN is particularly useful in resource-starved and distribution-shifted scenarios.

Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation

“…Although we learnt about mechanics involved in idiomatic translations, the vast majority of translations was still word for word, indicating that noncompositional processing does not emerge well (enough) in Transformer. Paradoxically, a recent trend is to encourage more compositional processing in NMT (Chaabouni et al, 2021;Li et al, 2021;Raunak et al, 2019, i.a.). We recommend caution since this inductive bias may harm idiom translations.…”

“…These patterns are stronger for figurative PIEs that the model paraphrases than for sentences that receive an overly compositional translation and hold across the seven European languages. Considering that a recent trend in NLP is to encourage even more compositional processing in NMT (Raunak et al, 2019;Chaabouni et al, 2021;Li et al, 2021, i.a. ), we recommend caution.…”

Improving Systematic Generalization Through Modularity and Augmentation

“…Ruis, Burghouts, & Bucur, 2021), natural language processing Baroni, 2020;Keysers et al, 2020;Kim & Linzen, 2020), and more generally (Nam & McClelland, 2021). Two fundamentally different approaches are taken by the literature; one utilizes additional data while making few changes to the conventional setup and architecture (Furrer, van Zee, Scales, & Schärli, 2020), while the other utilizes additional inductive biases that aim to support systematic generalization (Russin et al, 2019;Lake, 2019;Andreas, 2020;Nye et al, 2020;Gordon et al, 2020;Bogin et al, 2021;Chaabouni, Dessì, & Kharitonov, 2021). In this work we apply both approaches, the former through data augmentation, and the latter through high-level modularity.…”

State-of-the-art generalisation research in NLP: A taxonomy and review

“…The first prominent type of generalisation that can be found in the literature is compositional generalisation, which is often argued to underpin human's ability to quickly generalise to new data, tasks and domains (Fodor and Pylyshyn, 1988;Lake et al, 2017;Marcus, 2018;Schmidhuber, 1990). Because of this strong connection with humans and human language, work on compositional generalisation often has a primarily cognitive motivation, although practical concerns such as sample efficiency, quick adaptation and good generalisation in low-resource scenarios are frequently mentioned as additional or alternative motivations (Chaabouni et al, 2021;Linzen, 2020, to give just a few examples). While it has a strong intuitive appeal and clear mathematical definition (Montague, 1970), compositional generalisation is not easy to pin down empirically.…”