Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks

McCoy, R. Thomas; Frank, Robert; Linzen, Tal

doi:10.1162/tacl_a_00304

Cited by 52 publications

(70 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…C Encoder-Decoder architecture and its mathematics In order to model the production of plurals for any given Maltese singulars we created a deep learning model in the form of an encoder-decoder neural network. This network consists of two neural networks, an encoder and a decoder (McCoy et al, 2020). These are fed singulars and plurals respectively.…”

Section: Discussionmentioning

confidence: 99%

“…To investigate the production of Maltese plurals for any given singular, we used an encoder-decoder neural network (McCoy et al, 2020). We used an encoderdecoder neural network, rather than the existing Minimal Generalization Learner (MGL) proposed by Albright and Hayes (2003), because the MGL cannot model non-concatenative morphology for principled reasons.…”

Section: The Present Studymentioning

confidence: 99%

“…After this, we turn to modeling the production of plurals for any given singular. We do so in an encoder-decoder network (McCoy et al, 2020). The network is able to produce correct, or very similar to correct plurals.…”

Section: The Present Studymentioning

confidence: 99%

“…Models, which can handle non-local changes, and that are recently going through a revival, are connectionist deep learning models (McClelland and Patterson, 2002;McCoy et al, 2020;Pater, 2019;Prickett et al, 2018). 2 Recently, these models were used to model learning and generalisation of syntactic structures (McCoy et al, 2020), especially focusing on the question of possible hierarchical biases that are necessary prerequisites for learning syntax. They are also used to argue that reduplication can be learned without abstract variables (Prickett et al, 2018).…”

Section: Decoder Networkmentioning

confidence: 99%

“…We tested two different architectures that have been used for linguistic phenomena: Long short-term memory (LSTM) and gated recurrent units (GRU) (McCoy et al, 2020). Each architecture was further tested with an attention layer and without an attention layer.…”

Section: Decoder Networkmentioning

confidence: 99%

See 4 more Smart Citations

Modeling Maltese Noun Plural Classes without Morphemes

Nieder¹,

Tomaschek²,

Cohrs³

et al. 2020

Preprint

View full text Add to dashboard Cite

Word and Paradigm morphology proposes that morphologically complex words are stored as wholes in the mental lexicon, and by doing so it avoids problems that are related to the notion of the morpheme. However, it is not yet clear to what degree it is possible to computationally model classification and production of complex word forms without morphemes.We take up this question by modeling the classification and production of the Maltese noun plural system. Maltese is a Semitic language that has two broad classes of plurals: concatenative ones and non-concatenative ones. We model the classification in two models, and the production in a third one. The first model for classification, the Tilburg Memory Based Learner (TiMBL), is a computational implementation of exemplar models. TiMBL is impartial as to the existence of morphemes in the mental lexicon, and allows us to directly compare the classification with and without morphemes. It turns out that the classification with and without morphemes of Maltese nouns is equally good. The second classification model is the Naive Discriminative Learner (NDL). NDL is a computational implementation of discriminative learning. It can be understood as an implementation of the Word and Paradigm model and thus eschews morphemes. It differs from TiMBL in its assumptions about learning, and they way in which the classifications are obtained. NDL, too, classifies Maltese nouns well. A classification task is very different from a production task, and therefore we used neural networks to model the production of plurals. In these models we address the question whether the production of a plural noun for a given singular can be modeled without recourse to morphemes. We used two neural networks architectures (LSTM and GRU) that have been applied to linguistic phenomena, and find that they are able to correctly produce plurals, without making use of morphemes.We conclude that the Maltese noun plural system can be modeled on the basis of whole words alone without morphemes. These results, therefore, support the Word and Paradigm theory of the mental lexicon.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: The Present Studymentioning

confidence: 99%

Section: The Present Studymentioning

confidence: 99%

Section: Decoder Networkmentioning

confidence: 99%

Section: Decoder Networkmentioning

confidence: 99%

See 3 more Smart Citations

Modeling Maltese Noun Plural Classes without Morphemes

Nieder¹,

Tomaschek²,

Cohrs³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

Procedural Strategies

Beinborn,

Hollenstein

2023

Synthesis Lectures on Human Language Technologies

View full text Add to dashboard Cite

The ambiguity of BERTology: what do large language models represent?

Buder-Gröndahl

2023

Synthese

View full text Add to dashboard Cite

The field of “BERTology” aims to locate linguistic representations in large language models (LLMs). These have commonly been interpreted as representing structural descriptions (SDs) familiar from theoretical linguistics, such as abstract phrase-structures. However, it is unclear how such claims should be interpreted in the first place. This paper identifies six possible readings of “linguistic representation” from philosophical and linguistic literature, concluding that none has a straight-forward application to BERTology. In philosophy, representations are typically analyzed as cognitive vehicles individuated by intentional content. This clashes with a prevalent mentalist interpretation of linguistics, which treats SDs as (narrow) properties of cognitive vehicles themselves. I further distinguish between three readings of both kinds, and discuss challenges each brings for BERTology. In particular, some readings would make it trivially false to assign representations of SDs to LLMs, while others would make it trivially true. I illustrate this with the concrete case study of structural probing: a dominant model-interpretation technique. To improve the present situation, I propose that BERTology should adopt a more “LLM-first” approach instead of relying on pre-existing linguistic theories developed for orthogonal purposes.

show abstract

Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks

Cited by 52 publications

References 32 publications

Modeling Maltese Noun Plural Classes without Morphemes

Modeling Maltese Noun Plural Classes without Morphemes

Procedural Strategies

The ambiguity of BERTology: what do large language models represent?

Contact Info

Product

Resources

About