Learning compositionally through attentive guidance

Abstract

While neural network models have been successfully applied to domains that require substantial generalisation skills, recent studies have implied that they struggle when solving the task they are trained on requires inferring its underlying compositional structure. In this paper, we introduce Attentive Guidance, a mechanism to direct a sequence to sequence model equipped with attention to find more compositional solutions. We test it on two tasks, devised precisely to assess the compositional capabilities of neural models, and we show that vanilla sequence to sequence models with attention overfit the training distribution, while the guided versions come up with compositional solutions that fit the training and testing distributions almost equally well. Moreover, the learned solutions generalise even in cases where the training and testing distributions strongly diverge. In this way, we demonstrate that sequence to sequence models are capable of finding compositional solutions without requiring extra components. These results helps to disentangle the causes for the lack of systematic compositionality in neural networks, which can in turn fuel future work.

Transcoding Compositionally: Using Attention to Find More Generalizable Solutions

“…Our method draws inspiration from the work on compositional learning of Hupkes et al (2018a). The authors introduce the concept of Attentive Guidance, a training signal given to the attention mechanism of a seq2seq model to induce more compositional solutions.…”

“…We borrow the setup presented in Hupkes et al (2018a), which differs slightly from the setup as it was originally presented. In this setup, a typical input output example could be 001 t1 t2 → 001 010 111.…”

“…The remaining 2 held-out inputs are used to test for generalization. Following Hupkes et al (2018a), we do not include all 64 binary compositions in the training set, but leave out some for testing. In particular, we create one test set that contains all binary compositions containing t7 or t8, which are thus only seen in the training set as unary compositions.…”

Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN

“…Our findings indicate the following. Firstly, as hypothesized before (Dessì and Baroni, 2019;Hupkes et al, 2018), the limited attention span provides a useful inductive bias that allows models to perform better on compositional generalization induction, that SCAN probes for. Further, endowing a model with SCAN-style generalization capabilities can lead to improvements in low-resource and distribution-shifted scenarios as long as we ensure that we do not overfit to SCAN.…”

On the Interplay Between Fine-tuning and Composition in Transformers

“…Phrase and sentence composition has drawn frequent attention in analysis of neural models, often focusing on analysis of internal representations and downstream task behavior (Ettinger et al, 2018;Conneau et al, 2019;Nandakumar et al, 2019;Yu and Ettinger, 2020;Bhathena et al, 2020;Mu and Andreas, 2020; 1 Datasets and code available at https://github.com/yulang/fine-tuning-and-compositionin-transformers Andreas, 2019). Some work investigates compositionality via constructing linguistic (Keysers et al, 2019) and non-linguistic (Liška et al, 2018;Hupkes et al, 2018;Baan et al, 2019) synthetic datasets.…”