Computation in recurrent neural networks: From counters to iterated function systems

Kalinke, Yvonne; Lehmann, Helko

doi:10.1007/bfb0095051

Cited by 8 publications

(8 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A range of studies has experimentally examined the ability of recurrent networks to model counter languages such as a n b n (Kalinke and Lehmann, 1998;Gers and Schmidhuber, 2001;Cartling, 2008;Weiss et al, 2018;Suzgun et al, 2019). Other work has experimentally studied the performance of recurent architectures on learning to recognize well-bracketed strings, a similar but more challenging problem (Sennhauser and Berwick, 2018;Skachkova et al, 2018;Bernardy, 2018).…”

Section: Investigating the Power Of Sequence Modelingmentioning

confidence: 99%

“…Hierarchical structure is widely thought to be essential to modeling natural language, in particular its syntax (Everaert et al, 2015). Consequently, many researchers have studied the capability of recurrent neural network models to capture context-free languages (e.g., Kalinke and Lehmann, 1998;Gers and Schmidhuber, 2001;Grüning, 2006;Weiss et al, 2018;Sennhauser and Berwick, 2018;Korsky and Berwick, 2019) and linguistic phenomena involving hierarchical structure (e.g., Linzen et al, 2016;Gulordava et al, 2018). Some experimental evidence suggests that transformers might not be as strong as LSTMs at modeling hierarchical structure (Tran et al, 2018), though analysis studies have shown that transformer-based models encode a good amount of syntactic knowledge (e.g., Clark et al, 2019;Lin et al, 2019;Tenney et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Theoretical Limitations of Self-Attention in Neural Sequence Models

Hahn

2020

Transactions of the Association for Computational Linguistics

104

View full text Add to dashboard Cite

Transformers are emerging as the new workhorse of NLP, showing great success across tasks. Unlike LSTMs, transformers process input sequences entirely through self-attention. Previous work has suggested that the computational capabilities of self-attention to process hierarchical structures are limited. In this work, we mathematically investigate the computational power of self-attention to model formal languages. Across both soft and hard attention, we show strong theoretical limitations of the computational abilities of self-attention, finding that it cannot model periodic finite-state languages, nor hierarchical structure, unless the number of layers or heads increases with input length. These limitations seem surprising given the practical success of self-attention and the prominent role assigned to hierarchical structure in linguistics, suggesting that natural language can be approximated well with models that are too weak for the formal languages typically assumed in theoretical linguistics.

show abstract

Section: Investigating the Power Of Sequence Modelingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Theoretical Limitations of Self-Attention in Neural Sequence Models

Hahn

2020

Transactions of the Association for Computational Linguistics

104

View full text Add to dashboard Cite

show abstract

“…Linzen et al (2016) study the application of LSTMs to certain grammatical phenomena. RNNs and their variants have been used for recognizing Dyck words (Kalinke and Lehmann, 1998;Deleu and Dureau, 2016). Li et al (2017) evaluate their nonlinear weighted finite automata model on a Dyck language.…”

Section: Related Workmentioning

confidence: 99%

Closing Brackets with Recurrent Neural Networks

Skachkova¹,

Trost²,

Klakow³

2018

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP

View full text Add to dashboard Cite

Many natural and formal languages contain words or symbols that require a matching counterpart for making an expression wellformed. The combination of opening and closing brackets is a typical example of such a construction. Due to their commonness, the ability to follow such rules is important for language modeling. Currently, recurrent neural networks (RNNs) are extensively used for this task. We investigate whether they are capable of learning the rules of opening and closing brackets by applying them to synthetic Dyck languages that consist of different types of brackets. We provide an analysis of the statistical properties of these languages as a baseline and show strengths and limits of Elman-RNNs, GRUs and LSTMs in experiments on random samples of these languages. In terms of perplexity and prediction accuracy, the RNNs get close to the theoretical baseline in most cases.

show abstract

“…Previous work showed that LSTM outperforms traditional RNN algorithms on numerous tasks involving real-valued or discrete inputs and targets [5], [7], including tasks that require to learn the rules of regular languages (RLs) describable by deterministic finite-state automata (DFA) [1], [2], [8], [21], [31]. Until now, however, it has remained unclear whether LSTMs superiority carries over to tasks involving context-free languages (CFLs), such as those discussed in the RNN literature [16], [17], [23]- [25], [28].…”

Section: Introductionmentioning

confidence: 99%

LSTM recurrent networks learn simple context-free and context-sensitive languages

Gers

Schmidhuber

2001

IEEE Trans. Neural Netw.

589

271

View full text Add to dashboard Cite

Previous work on learning regular languages from exemplary training sequences showed that long short-term memory (LSTM) outperforms traditional recurrent neural networks (RNNs). We demonstrate LSTMs superior performance on context-free language benchmarks for RNNs, and show that it works even better than previous hardwired or highly specialized architectures. To the best of our knowledge, LSTM variants are also the first RNNs to learn a simple context-sensitive language, namely a(n)b(n)c(n).

show abstract

Computation in recurrent neural networks: From counters to iterated function systems

Cited by 8 publications

References 5 publications

Theoretical Limitations of Self-Attention in Neural Sequence Models

Theoretical Limitations of Self-Attention in Neural Sequence Models

Closing Brackets with Recurrent Neural Networks

LSTM recurrent networks learn simple context-free and context-sensitive languages

Contact Info

Product

Resources

About