Can Recurrent Neural Networks Validate Usage-Based Theories of Grammar Acquisition?

Pannitto, Ludovica; Herbelot, Aurélie

doi:10.3389/fpsyg.2022.741321

Cited by 4 publications

(3 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From a usage-based viewpoint, Diessel (2013b) stresses the importance of deictic pointing and joint attention as (extralinguistic) language acquisition fac- The non-improvements added by the curriculum approach also further add to the debate on what language models mean for linguistic theory. For example, Pannitto and Herbelot (2022) and Piantadosi (2023) have stressed the anti-Chomskyan evidence provided by the successes of language models. Curriculum learning looks like an obvious choice when trying to implement usage-based findings in the training process for (smaller) language models.…”

Section: Discussionmentioning

confidence: 99%

GPT-wee: How Small Can a Small Language Model Really Get?

Bunzeck,

Zarrieß

2023

Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning

View full text Add to dashboard Cite

In this model report, we present an alternative approach to improving language models through scaling up their architectures and training data. In contrast, we train significantly smaller GPT-wee language models for the CMCL and CoNLL shared task: the BabyLM challenge. Drawing inspiration from usagebased linguistics, specifically focusing on language acquisition factors such as frequency, word length, and lexical frames, we also conduct tests employing curriculum learning techniques. Our findings demonstrate that even very small models can achieve considerable proficiency in standard evaluation tasks, performing as good as or even better than much larger baseline models, both on zero-shot evaluation and tasks that require further fine-tuning. Our naïve curriculum approach, however, does not show any straightforward improvements, except for certain, very specific tasks. Overall, the results remain inconclusive and suggest interaction effects between model architecture, data make-up and learning processes that warrant further inspection.

show abstract

Section: Discussionmentioning

confidence: 99%

GPT-wee: How Small Can a Small Language Model Really Get?

Bunzeck,

Zarrieß

2023

Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning

View full text Add to dashboard Cite

show abstract

“…RNN is more effective in learning nonlinear characteristics of sequences because they share parameters, are remembered, and are Turing complete. RNN is first used to describe the relationship between a sequence's current output and its past information [ 25 , 26 ].…”

Section: Models and Evaluation Methodsmentioning

confidence: 99%

Evaluation of the Practical Effects of Environmental Measures in the Conservation of Architectural Heritage in Yan’an Based on Recurrent Neural Networks

Wang

2022

Journal of Environmental and Public Health

View full text Add to dashboard Cite

Yan’an is one of the “two holy places” of the Chinese nation and the Chinese revolution and is one of the first cities of historical and cultural significance and an outstanding tourist city in China, as announced by the state council. The evaluation of the effectiveness of environmental conservation is one of the very important elements of the conservation of Yan’an’s architectural heritage. However, the existing evaluation methods cannot provide new solutions for decision-making, the meaning of the comprehensive evaluation function is unclear, the naming clarity is low, there is less quantitative data and more qualitative components, and the results are not easily convincing. This paper proposes a method for evaluating the practical effects of environmental class measures in the conservation of Yan’an’s architectural heritage based on recurrent neural networks. The recurrent neural network makes full use of the memory function in the network, considers the causal relationship of the actual effect, and efficiently evaluates the existing measures. In comparison with factor analysis and hierarchical analysis, this paper has greater applicability in evaluating the practical effects of environmental measures in the conservation of Yan’an’s architectural heritage and is basically consistent with the results of the theoretical analysis. It provides a scientific basis for the construction and implementation of environmental measures for the architectural heritage of Yan’an.

show abstract

“…On the other hand, deriving the linguistic structure from the input structure itself requires investigating the two aspects together. So far, most studies on NLMs have disregarded the effect of the input on experimental results (Pannitto and Herbelot, 2022).…”

Section: Nativist Vs Non-nativist Approaches To Language Acquisitionmentioning

confidence: 99%

Recurrent babbling: evaluating the acquisition of grammar from limited input data

Pannitto¹,

Herbelot²

2020

Proceedings of the 24th Conference on Computational Natural Language Learning

View full text Add to dashboard Cite

Recurrent Neural Networks (RNNs) have been shown to capture various aspects of syntax from raw linguistic input. In most previous experiments, however, learning happens over unrealistic corpora, which do not reflect the type and amount of data a child would be exposed to. This paper remedies this state of affairs by training a Long Short-Term Memory network (LSTM) over a realistically sized subset of child-directed input. The behaviour of the network is analysed over time using a novel methodology which consists in quantifying the level of grammatical abstraction in the model's generated output (its 'babbling'), compared to the language it has been exposed to. We show that the LSTM indeed abstracts new structures as learning proceeds.

show abstract

Can Recurrent Neural Networks Validate Usage-Based Theories of Grammar Acquisition?

Cited by 4 publications

References 43 publications

GPT-wee: How Small Can a Small Language Model Really Get?

GPT-wee: How Small Can a Small Language Model Really Get?

Evaluation of the Practical Effects of Environmental Measures in the Conservation of Architectural Heritage in Yan’an Based on Recurrent Neural Networks

Recurrent babbling: evaluating the acquisition of grammar from limited input data

Contact Info

Product

Resources

About