Recurrent Neural Networks With External Addressable Long-Term and Working Memory for Learning Long-Term Dependences

Quan, Zhibin; Zeng, Weili; Li, Xuelian; Liu, Yandong; Yu, Yunxiu; Yang, Wankou

doi:10.1109/tnnls.2019.2910302

Cited by 27 publications

(11 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On task 1, 12, 15, 20, it yields the same performance viewing the state-of-theart of nowadays which can be seen as the best performance among all these four models. There are the tasks for which CMemN2N achieves the second-best results, such as tasks 3,5,6,8,10,11,13,14,17,18.…”

Section: Resultsmentioning

confidence: 99%

“…The reason for end-to-end design is that memory networks proposed in [10] is difficult to train via backpropagation, and demands supervision at each layer of the neural network, so we have [3] to realize model which can be trained endto-end from input to output to guarantee the continuity of memory networks. We test the proposed architecture on the Facebook bAbI task 1 as in [3], [14], [24], which is a nulti-hop reasoning problem.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Convolutional End-to-End Memory Networks for Multi-Hop Reasoning

Yang

Fan

2019

IEEE Access

View full text Add to dashboard Cite

Machine reading and comprehension using differentiable reasoning models has recently been studied extensively, and memory networks have demonstrated promising performance on some reasoning tasks such as factual reasoning and basic deduction. However, as a natural language understanding model, memory networks still face challenges on the numeric representations for sentences, particularly the text representation method and the effectiveness of learned vector representations. In this paper, inspired by the convolution mechanism in the computer vision domain, a raw text representation architecture for question answering problem named convolutional end-to-end memory networks(CMemN2N) architecture is proposed. The convolutional architecture of the proposed model allows us to abstract the useful local information for reasoning to get the significant numeric sentence representation passed to the follow-up subtasks. Our experiments show that CMemN2N achieves better results on most of the 20 bAbI task dataset, yielding improvements for the average result compared to the state-of-the-art.INDEX TERMS End-to-end memory networks, convolutional network, natural language reasoning.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Convolutional End-to-End Memory Networks for Multi-Hop Reasoning

Yang

Fan

2019

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Specifically, by optimizing the connections of the inner LSTM cells for the performance enhancement of LSTMdominated networks, adding learnable nonlinear state-togate memory connections performs noticeably better than the vanilla LSTM for various tasks with longer sequences [79], while conducting convolutional operation on the two input-to-state/state-to-state transitions and on the previous outputs/current input of the LSTM can integrate long-term temporal dependence with time-frequency characteristics [80] and capture the contextual relationships of the data [81], respectively. Furthermore, by integrating LSTM networks with other components (such as graph neural networks [82] and external memory [83]), learning 3D contexts and the temporal dynamics of multiple studies can accurately estimate 4D changes [84], [85], while exploiting the frame-level dependencies with LSTM (or the shot-level dependencies with graph convolutional networks) [82] and remembering previous metaknowledge [86] in the optimization of performance across similarly structured tasks can perform key-shot [82] and one-shot learning [86], respectively. However, most memory mechanisms rely on weight-like storage (e.g., RNNs) or information-flow gating (e.g., LSTMs) rather than activity-based task-relevant information maintenance of working memory, which yields the best compressed transformative representation of dynamic environments for flexibility/generalizability across tasks [87].…”

Section: Working Memory Inspired Deep Learningmentioning

confidence: 99%

Working memory inspired hierarchical video decomposition with transformative representations

Qin¹,

Mao²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Video decomposition is very important to extract moving foreground objects from complex backgrounds in computer vision, machine learning, and medical imaging, e.g., extracting moving contrast-filled vessels from the complex and noisy backgrounds of X-ray coronary angiography (XCA). However, the challenges caused by dynamic backgrounds, overlapping heterogeneous environments and complex noises still exist in video decomposition. To solve these challenges, this study is the first to introduce a flexible visual working memory model in video decomposition to provide interpretable and high-performance hierarchical deep learning architecture, integrating the transformative representations between sensory and control layers from the perspective of visual and cognitive neuroscience. Specifically, robust PCA unrolling networks acting as a structure-regularized sensor layer decompose XCA into sparse/low-rank structured representations to separate moving contrast-filled vessels from noisy and complex backgrounds. Then, patch recurrent convolutional LSTM networks with a backprojection superresolution module embody unstructured random representations of the control layer in working memory, recurrently projecting spatiotemporally decomposed nonlocal patches into orthogonal subspaces for heterogeneous vessel retrieval and interference suppression. This video decomposition architecture effectively restores the heterogeneous profiles of intensity and geometry of moving objects against the complex background interferences. Experiments show that the proposed method significantly outperforms state-of-the-art methods in accurate moving contrast-filled vessel extraction with excellent flexibility and computational efficiency.

show abstract

“…The IndRNN has the capability to handle long-term dependence that it can even work on the sequence longer than 5000 time steps. Recently, there have also been efforts such as [26] to use additional mechanisms to assist RNN in learning long-term dependence.…”

Section: B Gate-free Recurrent Neural Networkmentioning

confidence: 99%

Learning Various Length Dependence by Dual Recurrent Neural Networks

Zhang¹,

Li²,

Ye³

et al. 2020

Preprint

View full text Add to dashboard Cite

Recurrent neural networks (RNNs) are widely used as a memory model for sequence-related problems. Many variants of RNN have been proposed to solve the gradient problems of training RNNs and process long sequences. Although some classical models have been proposed, capturing long-term dependence while responding to short-term changes remains a challenge. To this problem, we propose a new model named Dual Recurrent Neural Networks (DuRNN). The DuRNN consists of two parts to learn the short-term dependence and progressively learn the long-term dependence. The first part is a recurrent neural network with constrained full recurrent connections to deal with short-term dependence in sequence and generate short-term memory. Another part is a recurrent neural network with independent recurrent connections which helps to learn long-term dependence and generate long-term memory. A selection mechanism is added between two parts to help the needed long-term information transfer to the independent neurons. Multiple modules can be stacked to form a multi-layer model for better performance. Our contributions are: 1) a new recurrent model developed based on the divide-and-conquer strategy to learn long and short-term dependence separately, and 2) a selection mechanism to enhance the separating and learning of different temporal scales of dependence. Both theoretical analysis and extensive experiments are conducted to validate the performance of our model, and we also conduct simple visualization experiments and ablation analyses for the model interpretability. Experimental results indicate that the proposed DuRNN model can handle not only very long sequences (over 5000 time steps), but also short sequences very well. Compared with many state-of-the-art RNN models, our model has demonstrated efficient and better performance.

show abstract

Recurrent Neural Networks With External Addressable Long-Term and Working Memory for Learning Long-Term Dependences

Cited by 27 publications

References 23 publications

Convolutional End-to-End Memory Networks for Multi-Hop Reasoning

Convolutional End-to-End Memory Networks for Multi-Hop Reasoning

Working memory inspired hierarchical video decomposition with transformative representations

Learning Various Length Dependence by Dual Recurrent Neural Networks

Contact Info

Product

Resources

About