Generalising to German Plural Noun Classes, from the Perspective of a Recurrent Neural Network

Dankers, Verna; Langedijk, Anna; McCurdy, Kate; Williams, Adina; Hupkes, Dieuwke

doi:10.18653/v1/2021.conll-1.8

Cited by 8 publications

(5 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We plot this behavior for models trained on increasing amounts of epochs in Figure 1B, from which it can be seen that this task follows a peculiar inverse scaling pattern (McKenzie et al, 2023). Exploring this pattern in more detail could provide an interesting direction for future research, connecting it to the rule learning of irregular forms in LMs (Dankers et al, 2021).…”

Section: Resultsmentioning

confidence: 99%

ChapGTP, ILLC’s Attempt at Raising a BabyLM: Improving Data Efficiency by Automatic Task Formation

Jumelet,

Hanna,

de Heer Kloots

et al. 2023

Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning

View full text Add to dashboard Cite

We present the submission of the ILLC at the University of Amsterdam to the BabyLM challenge (Warstadt et al., 2023), in the strict-small track.Our final model, ChapGTP, is a masked language model that was trained for 200 epochs, aided by a novel data augmentation technique called Automatic Task Formation. We discuss in detail the performance of this model on the three evaluation suites: BLiMP, (Super)GLUE, and MSGS. Furthermore, we present a wide range of methods that were ultimately not included in the model, but may serve as inspiration for training LMs in low-resource settings.

show abstract

Section: Resultsmentioning

confidence: 99%

ChapGTP, ILLC’s Attempt at Raising a BabyLM: Improving Data Efficiency by Automatic Task Formation

Jumelet,

Hanna,

de Heer Kloots

et al. 2023

Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning

View full text Add to dashboard Cite

show abstract

“…The German plural system, on the other hand, relies on the addition of a regular suffix (/-e/, /-er/, /-en/, /-s/ or /-ø/) to the singular form which makes the modeling of pluralization process closer to a classification problem. Dankers et al (2021) employed a simple unidirectional recurrent encoder-decoder model based on Long-Short-Term-Memory (LSTM) layers with no attention mechanism, while the research reported by Beser (2021) compared the performance of a bidirectional LSTM model to a vanilla transformer model. The performance of both approaches relied immensely on the frequency of each suffix in a German corpus.…”

Section: Related Workmentioning

confidence: 99%

Neural Arabic singular-to-plural conversion using a pretrained Character-BERT and a fused transformer

Radman,

Atros,

Duwairi

2023

Nat. Lang. Eng.

View full text Add to dashboard Cite

Morphological re-inflection generation is one of the most challenging tasks in the natural language processing (NLP) domain, especially with morphologically rich, low-resource languages like Arabic. In this research, we investigate the ability of transformer-based models in the singular-to-plural Arabic noun conversion task. We start with pretraining a Character-BERT model on a masked language modeling task using 1,134,950 Arabic words and then adopting the fusion technique to transfer the knowledge gained by the pretrained model to a full encoder–decoder transformer model, in one of the proposed settings. The second proposed setting directly fuses the output Character-BERT embeddings into the decoder. We then analyze and compare the performance of the two architectures and provide an interpretability section in which we track the features of attention with respect to the model. We perform the interpretation on both the macro and micro levels, providing some individual examples. Moreover, we provide a thorough error analysis showing the strengths and weaknesses of the proposed framework. To the best of our knowledge, this is the first effort in the Arabic NLP domain that adopts the development of an end-to-end fused-transformer deep learning model to address the problem of singular-to-plural conversion.

show abstract

“…It has been argued that LMs acquire an abstract notion of word order that goes beyond mere n-gram co-occurrence statistics (Futrell and Levy, 2019;Kuribayashi et al, 2020;Merrill et al, 2024), a claim that we in this paper assess for large-scale LMs in the context of adjective order. Finally, numerous works have investigated the trade-off between memorization and generalization in LMs: it has been shown that larger LMs are able to memorize entire passages from the training data (Biderman et al, 2023a;Lesci et al, 2024;Prashanth et al, 2024), but generalization patterns for grammatical phenomena have also been shown to follow human-like generalization (Dankers et al, 2021;Hupkes et al, 2023;Alhama et al, 2023).…”

Section: Word Order In Language Modelsmentioning

confidence: 99%

Feature Interactions Reveal Linguistic Structure in Language Models

Jumelet¹,

Zuidema²

2023

Findings of the Association for Computational Linguistics: ACL 2023

View full text Add to dashboard Cite

In English and other languages, multiple adjectives in a complex noun phrase show intricate ordering patterns that have been a target of much linguistic theory. These patterns offer an opportunity to assess the ability of language models (LMs) to learn subtle rules of language involving factors that cross the traditional divisions of syntax, semantics, and pragmatics. We review existing hypotheses designed to explain Adjective Order Preferences (AOPs) in humans and develop a setup to study AOPs in LMs: we present a reusable corpus of adjective pairs and define AOP measures for LMs. With these tools, we study a series of LMs across intermediate checkpoints during training. We find that all models' predictions are much closer to human AOPs than predictions generated by factors identified in theoretical linguistics. At the same time, we demonstrate that the observed AOPs in LMs are strongly correlated with the frequency of the adjective pairs in the training data and report limited generalization to unseen combinations. This highlights the difficulty in establishing the link between LM performance and linguistic theory. We therefore conclude with a road map for future studies our results set the stage for, and a discussion of key questions about the nature of knowledge in LMs and their ability to generalize beyond the training sets. 1

show abstract

Generalising to German Plural Noun Classes, from the Perspective of a Recurrent Neural Network

Cited by 8 publications

References 52 publications

ChapGTP, ILLC’s Attempt at Raising a BabyLM: Improving Data Efficiency by Automatic Task Formation

ChapGTP, ILLC’s Attempt at Raising a BabyLM: Improving Data Efficiency by Automatic Task Formation

Neural Arabic singular-to-plural conversion using a pretrained Character-BERT and a fused transformer

Feature Interactions Reveal Linguistic Structure in Language Models

Contact Info

Product

Resources

About