Separation of Memory and Processing in Dual Recurrent Neural Networks

Oliva, Christian; Lago-Fernández, Luis F.

doi:10.1007/978-3-030-86380-7_29

Cited by 3 publications

(3 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The dual architecture was firstly proposed as an alternative that reduces the computational load on the recurrent layer, letting it concentrate on modeling the temporal dependencies only. From a more abstract point of view, it has been argued that the dual architecture can be understood as a sort of Mealy machine, where the output explicitly depends on both the hidden state and the input [16]. Our results show that this explicit dependence on the input can indeed lead to better performance on language modeling tasks.…”

Section: Discussionmentioning

confidence: 63%

“…In this work, we have presented a new network design for the Language Modeling task based on the dual network proposed by [16]. This network adds a direct connection between the input and the output, skipping the recurrent module, and can be adapted to any of the traditional Embedding-Recurrent-Softmax (ERS) models, opening the way to new approaches for this task.…”

Section: Discussionmentioning

confidence: 99%

“…In this work we use moderate-scale language modeling datasets to explore the effect of a mechanism recently proposed by [16], when combined with different LSTM-based models in the language modeling context. The idea consists of modifying a recurrent architecture by introducing a direct connection between the input and the output of the recurrent module.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

The Importance of the Current Input in Sequence Modeling

Oliva¹,

Lago-Fernández²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

The last advances in sequence modeling are mainly based on deep learning approaches. The current state of the art involves the use of variations of the standard LSTM architecture, combined with several tricks that improve the final prediction rates of the trained neural networks. However, in some cases, these adaptations might be too much tuned to the particular problems being addressed. In this article, we show that a very simple idea, to add a direct connection between the input and the output, skipping the recurrent module, leads to an increase of the prediction accuracy in sequence modeling problems related to natural language processing. Experiments carried out on different problems show that the addition of this kind of connection to a recurrent network always improves the results, regardless of the architecture and training-specific details. When this idea is introduced into the models that lead the field, the resulting networks achieve a new state-of-the-art perplexity in language modeling problems.

show abstract

Section: Discussionmentioning

confidence: 63%

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation