Context-Free Transductions with Neural Stacks

Hao, Yiding; Merrill, William; Angluin, Dana; Frank, Robert; Amsel, Noah; Benz, Andrew; Mendelsohn, Simon

doi:10.48550/arxiv.1809.02836

Cited by 6 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For these reasons, we think of the D >1 languages as expressing the core of what it means to be a contextfree language with hierarchical structure, even if it is not itself a universal CFL. This property of the Dyck languages accounts for the heavy focus on the them in prior work (Deleu and Dureau, 2016;Bernardy, 2018;Sennhauser and Berwick, 2018;Skachkova et al, 2018;Hao et al, 2018;Zaremba et al, 2016;Suzgun et al, 2019a;Yu et al, 2019;Hahn, 2019) as well as in this work. It would thus be notable for finite precision neural networks to learn languages, like the D >1 languages and other languages requiring a stack, if we want these neural architectures to be able to manifest hierarchical structures.…”

Section: Introductionsupporting

confidence: 63%

“…In contrast to the models in Joulin and Mikolov (2015); Grefenstette et al (2015); Hao et al (2018); Yu et al (2019), our architectures are economical: Unless otherwise stated, the models are all single-layer networks with 8 hidden units. In all the experiments, the entries of the memory were set to be one-dimensional, while the size of the memory in the Baby-NTMs was fixed to 104 (since the length of the longest sequence in all the tasks was 100).…”

Section: Training Detailsmentioning

confidence: 93%

“…However, as the author mentions, the specificity of the RUSS architecture disqualifies it as a practical model choice for real-world language modeling tasks. Hao et al (2018) studied the interpretability of Neural Stack models (Grefenstette et al, 2015) in a number of simple language modeling tasks, including parenthesis prediction, string reversal, and XOR evaluation. Though their Neural Stacks exhibited intuitive stack behaviors on their contextfree transduction tasks and performed almost as well as the standard LSTM models, the authors noted that their stack-augmented models were more difficult to train than the traditional LSTMs.…”

Section: Investigations Of the Dyck Languagesmentioning

confidence: 99%

“…In this paper, we introduce three enhanced RNN models that consist of recurrent layers and external memory structures, namely stack-augmented RNNs (Stack-RNNs), stack-augmented LSTMs (Stack-LSTMs), and Baby Neural Turing Machines (Baby-NTMs), and show that they can effectively learn to recognize some D >1 languages from limited data by emulating deterministic pushdown automata. Previous studies used simple RNN models (Bernardy, 2018;Sennhauser and Berwick, 2018;Suzgun et al, 2019b;Yu et al, 2019) and memory-augmented architectures (Hao et al, 2018) to attempt to learn D 2 under different training platforms; however, none of them were able to obtain good performance on this task. We thus present the first demonstration that a memoryaugmented neural network (MARNN) can learn D >1 languages.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languages

Suzgun,

Gehrmann,

Belinkov

et al. 2019

Preprint

View full text Add to dashboard Cite

We introduce three memory-augmented Recurrent Neural Networks (MARNNs) and explore their capabilities on a series of simple language modeling tasks whose solutions require stack-based mechanisms. We provide the first demonstration of neural networks recognizing the generalized Dyck languages, which express the core of what it means to be a language with hierarchical structure. Our memory-augmented architectures are easy to train in an end-to-end fashion and can learn the Dyck languages over as many as six parenthesis-pairs, in addition to two deterministic palindrome languages and the string-reversal transduction task, by emulating pushdown automata. Our experiments highlight the increased modeling capacity of memory-augmented models over simple RNNs, while inflecting our understanding of the limitations of these models.1 From an automata-theoretic perspective, such languages are expressible with simple one-turn counter machines.

show abstract

Section: Introductionsupporting

confidence: 63%

Section: Training Detailsmentioning

confidence: 93%

Section: Investigations Of the Dyck Languagesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languages

Suzgun,

Gehrmann,

Belinkov

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Finally, observing the defects of both LSTM and Transformer in learning CFG and algorithmic tasks, many works propose to use external memory to enhance the LSTM model (Joulin and Mikolov 2015;Das, Giles, and Sun 1992;Suzgun et al 2019b), introduce recurrence in the Transformer network (Dehghani et al 2018), and design specialized architectures (Graves, Wayne, and Danihelka 2014;Hao et al 2018;Sukhbaatar et al 2015;Stogin et al 2020). Though it's commonly believed that LSTM with finite memory, i.e.…”

Section: Related Workmentioning

confidence: 99%

Learning Bounded Context-Free-Grammar via LSTM and the Transformer:Difference and Explanations

Shi¹,

Gao²,

Tian³

et al. 2021

Preprint

View full text Add to dashboard Cite

Long Short-Term Memory (LSTM) and Transformers are two popular neural architectures used for natural language processing tasks. Theoretical results show that both are Turingcomplete and can represent any context-free language (CFL). In practice, it is often observed that Transformer models have better representation power than LSTM. But the reason is barely understood. We study such practical differences between LSTM and Transformer and propose an explanation based on their latent space decomposition patterns. To achieve this goal, we introduce an oracle training paradigm, which forces the decomposition of the latent representation of LSTM and the Transformer, and supervises with the transitions of the Pushdown Automaton (PDA) of the corresponding CFL. With the forced decomposition, we show that the performance upper bounds of LSTM and Transformer in learning CFL are close: both of them can simulate a stack and perform stack operation along with state transitions. However, the absence of forced decomposition leads to the failure of LSTM models to capture the stack and stack operations, while having a marginal impact on the Transformer model. Lastly, we connect the experiment on the prototypical PDA to a real-world parsing task to re-verify the conclusions. 1

show abstract

A Neural State Pushdown Automata

Mali

Ororbia

Giles

2020

IEEE Trans. Artif. Intell.

View full text Add to dashboard Cite

In order to learn complex grammars, recurrent neural networks (RNNs) require sufficient computational resources to ensure correct grammar recognition. A widely-used approach to expand model capacity would be to couple an RNN to an external memory stack. Here, we introduce a "neural state" pushdown automaton (NSPDA), which consists of a digital stack, instead of an analog one, that is coupled to a neural network state machine. We empirically show its effectiveness in recognizing various context-free grammars (CFGs). First, we develop the underlying mechanics of the proposed higher order recurrent network and its manipulation of a stack as well as how to stably program its underlying pushdown automaton (PDA) to achieve desired finite-state network dynamics. Next, we introduce a noise regularization scheme for higher-order (tensor) networks, to our knowledge the first of its kind, and design an algorithm for improved incremental learning. Finally, we design a method for inserting grammar rules into a NSPDA and empirically show that this prior knowledge improves its training convergence time by an order of magnitude and, in some cases, leads to better generalization. The NSPDA is also compared to a classical analog stack neural network pushdown automaton (NNPDA) as well as a wide array of first and second-order RNNs with and without external memory, trained using different learning algorithms. Our results show that, for Dyck(2) languages, prior rule-based knowledge is critical for optimization convergence and for ensuring generalization to longer sequences at test time. We observe that many RNNs with and without memory, but no prior knowledge, fail to converge and generalize poorly on CFGs.

show abstract

Context-Free Transductions with Neural Stacks

Cited by 6 publications

References 0 publications

Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languages

Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languages

Learning Bounded Context-Free-Grammar via LSTM and the Transformer:Difference and Explanations

A Neural State Pushdown Automata

Contact Info

Product

Resources

About