Out of Order: How important is the sequential order of words in a sentence in Natural Language Understanding tasks?

Pham, Thang M.; Bui, Trung; Mai, Long; Nguyen, Anh

doi:10.18653/v1/2021.findings-acl.98

Cited by 43 publications

(44 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This performance gap closes appreciably as we perform more structured syntactic shifts such as reversing the sentence (a drop of 10%), or systematically permuting word orders using the dependency tree (a drop of between 7% and 9%). Rather than being invariant to word orders across natural language understanding tasks (Sinha et al, 2021;Pham et al, 2021), we instead find that BERTbased models are in fact sensitive to word order, at least for the tasks in the GLUE benchmark. In addition, we find that continued pretraining can close the performance gap to all but a few percent-age points for tree-based structural shifts.…”

Section: Syntax Matters But Not Too Muchcontrasting

confidence: 60%

“…While syntax is an crucial aspect of language, studies have also shown syntactic typology to be surprisingly non-predictive of transfer quality (Pham et al, 2021), and other studies have shown LLMs to be largely word-order invariant (Sinha et al, 2021). We investigate a set of syntactic transformations that isolate syntactic word-order shifts from the other factors that can vary between languages such as tokenization, static embeddings, and morphological representation.…”

Section: Syntactic Shiftsmentioning

confidence: 99%

See 1 more Smart Citation

Oolong: Investigating What Makes Crosslingual Transfer Hard with Controlled Studies

Wu¹,

Papadimitriou²,

Tamkin³

2022

Preprint

View full text Add to dashboard Cite

Little is known about what makes crosslingual transfer hard, since factors like tokenization, morphology, and syntax all change at once between languages. To disentangle the impact of these factors, we propose a set of controlled transfer studies: we systematically transform GLUE tasks to alter different factors one at a time, then measure the resulting drops in a pretrained model's downstream performance. In contrast to prior work suggesting little effect from syntax on knowledge transfer, we find significant impacts from syntactic shifts (3-6% drop), though models quickly adapt with continued pretraining on a small dataset. However, we find that by far the most impactful factor for crosslingual transfer is the challenge of aligning the new embeddings with the existing transformer layers (18% drop), with little additional effect from switching tokenizers (<2% drop) or word morphologies (<2% drop). Moreover, continued pretraining with a small dataset is not very effective at closing this gap-suggesting that new directions are needed for solving this problem.

show abstract

Section: Syntax Matters But Not Too Muchcontrasting

confidence: 60%

Section: Syntactic Shiftsmentioning

confidence: 99%

Oolong: Investigating What Makes Crosslingual Transfer Hard with Controlled Studies

Wu¹,

Papadimitriou²,

Tamkin³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Circumstantial evidence for the redundancy of word order comes from work such as that of Niven and Kao (2019), which showed that language models' predictions in certain tasks are largely explained by word-level triggers. Concurrently with this work, Sinha et al (2021a,b); Pham et al (2021); Gupta et al (2021) probed and demonstrated, in various ways, the surprising insensitivity of infilling LMs' performance on GLUE tasks to word order in training and evaluation data. These studies complement our discovery that nearly all of models' accuracy on GLUE tasks can be explained by bags of words only ( §5.2) to show that word order rarely carries information useful for classifying textual similarity, entailment, or sentiment.…”

Section: Related Workmentioning

confidence: 59%

Studying word order through iterative shuffling

Malkin¹,

Lanka²,

Goel³

et al. 2021

Preprint

View full text Add to dashboard Cite

As neural language models approach human performance on NLP benchmark tasks, their advances are widely seen as evidence of an increasingly complex understanding of syntax. This view rests upon a hypothesis that has not yet been empirically tested: that word order encodes meaning essential to performing these tasks. We refute this hypothesis in many cases: in the GLUE suite and in various genres of English text, the words in a sentence or phrase can rarely be permuted to form a phrase carrying substantially different information. Our surprising result relies on inference by iterative shuffling (IBIS), a novel, efficient procedure that finds the ordering of a bag of words having the highest likelihood under a fixed language model. IBIS can use any black-box model without additional training and is superior to existing word ordering algorithms. Coalescing our findings, we discuss how shuffling inference procedures such as IBIS can benefit language modeling and constrained generation.

show abstract

“…There is substantial evidence that RoBERTa is able to associate abstract constructional templates with their meaning without lexical cues. This result is perhaps surprising, given that previous work found that LMs are relatively insensitive to word order in compositional phrases (Yu and Ettinger, 2020) and downstream inference tasks (Sinha et al, 2021;Pham et al, 2021), where their performance can be largely attributed to lexical overlap.…”

Section: Potential Confoundsmentioning

confidence: 77%

Neural reality of argument structure constructions

Li¹,

Zhu²,

Thomas³

et al. 2022

Preprint

View full text Add to dashboard Cite

In lexicalist linguistic theories, argument structure is assumed to be predictable from the meaning of verbs. As a result, the verb is the primary determinant of the meaning of a clause. In contrast, construction grammarians propose that argument structure is encoded in constructions (or form-meaning pairs) that are distinct from verbs. Decades of psycholinguistic research have produced substantial empirical evidence in favor of the construction view. Here we adapt several psycholinguistic studies to probe for the existence of argument structure constructions (ASCs) in Transformerbased language models (LMs). First, using a sentence sorting experiment, we find that sentences sharing the same construction are closer in embedding space than sentences sharing the same verb. Furthermore, LMs increasingly prefer grouping by construction with more input data, mirroring the behaviour of non-native language learners. Second, in a "Jabberwocky" priming-based experiment, we find that LMs associate ASCs with meaning, even in semantically nonsensical sentences. Our work offers the first evidence for ASCs in LMs and highlights the potential to devise novel probing methods grounded in psycholinguistic research.

show abstract

Out of Order: How important is the sequential order of words in a sentence in Natural Language Understanding tasks?

Cited by 43 publications

References 35 publications

Oolong: Investigating What Makes Crosslingual Transfer Hard with Controlled Studies

Oolong: Investigating What Makes Crosslingual Transfer Hard with Controlled Studies

Studying word order through iterative shuffling

Neural reality of argument structure constructions

Contact Info

Product

Resources

About