On the Realization of Compositionality in Neural Networks

Joris, Baan,; Leible, Jana; Nikolaus, Mitja; Rau, D.; Ulmer, Dennis; Baumgärtner, Tim; Hupkes, Dieuwke; Bruni, Elia

doi:10.48550/arxiv.1906.01634

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These generalization tasks pose particular challenges, as they require generating OOD output sequences, composed of novel combinations of in-distribution output subsequences, from in-distribution input sequences. Whereas other approaches have focused on modifying the training objective (Lake, 2019;Baan et al, 2019), model architecture (Kuo et al, 2020;Gao et al, 2020), or data augmentation (Andreas, 2019;Akyürek et al, 2020), we use RD to reformulate and decompose a complex sequence generation task into a series of smaller predictions that appear in-distribution from the perspective of the model.…”

Section: Discussionmentioning

confidence: 99%

Recursive Decoding: A Situated Cognition Approach to Compositional Generation in Grounded Language Understanding

Setzler¹,

Howland²,

Phillips³

2022

Preprint

View full text Add to dashboard Cite

Compositional generalization is a troubling blind spot for neural language models. Recent efforts have presented techniques for improving a model's ability to encode novel combinations of known inputs, but less work has focused on generating novel combinations of known outputs. Here we focus on this latter "decode-side" form of generalization in the context of gSCAN, a synthetic benchmark for compositional generalization in grounded language understanding. We present Recursive Decoding (RD), a novel procedure for training and using seq2seq models, targeted towards decode-side generalization. Rather than generating an entire output sequence in one pass, models are trained to predict one token at a time. Inputs (i.e., the external gSCAN environment) are then incrementally updated based on predicted tokens, and re-encoded for the next decoder time step. RD thus decomposes a complex, out-ofdistribution sequence generation task into a series of incremental predictions that each resemble what the model has already seen during training. RD yields dramatic improvement on two previously neglected generalization tasks in gSCAN. We provide analyses to elucidate these gains over failure of a baseline, and then discuss implications for generalization in naturalistic grounded language understanding, and seq2seq more generally.

show abstract