Language Models as an Alternative Evaluator of Word Order Hypotheses: A Case Study in Japanese

Abstract

We examine a methodology using neural language models (LMs) for analyzing the word order of language. This LM-based method has the potential to overcome the difficulties existing methods face, such as the propagation of preprocessor errors in count-based methods. In this study, we explore whether the LMbased method is valid for analyzing the word order. As a case study, this study focuses on Japanese due to its complex and flexible word order. To validate the LM-based method, we test (i) parallels between LMs and human word order preference, and (ii) consistency of the results obtained using the LM-based method with previous linguistic studies. Through our experiments, we tentatively conclude that LMs display sufficient word order knowledge for usage as an analysis tool. Finally, using the LMbased method, we demonstrate the relationship between the canonical word order and topicalization, which had yet to be analyzed by largescale experiments.

Feature Interactions Reveal Linguistic Structure in Language Models

“…The importance of word order in LMs has been a topic of debate, with various works claiming that downstream performance is not affected by scrambled inputs (Malkin et al, 2021;Sinha et al, 2021), although it has been shown that LMs are able to retain a notion of word order through their positional embeddings (Abdou et al, 2022). It has been argued that LMs acquire an abstract notion of word order that goes beyond mere n-gram co-occurrence statistics (Futrell and Levy, 2019;Kuribayashi et al, 2020;Merrill et al, 2024), a claim that we in this paper assess for large-scale LMs in the context of adjective order. Finally, numerous works have investigated the trade-off between memorization and generalization in LMs: it has been shown that larger LMs are able to memorize entire passages from the training data (Biderman et al, 2023a;Lesci et al, 2024;Prashanth et al, 2024), but generalization patterns for grammatical phenomena have also been shown to follow human-like generalization (Dankers et al, 2021;Hupkes et al, 2023;Alhama et al, 2023).…”