Time Adaptive Recurrent Neural Network

Kag, Anil; Saligrama, Venkatesh

doi:10.1109/cvpr46437.2021.01490

Cited by 16 publications

(25 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This superior performance is further illustrated in Table 7, where the test perplexity for different models on the PTB word-level task is presented. We observe that not only does LEM significantly outperform (by around 40%) LSTM, but it also provides again the best performance among all single layer recurrent models, including the recently proposed TARNN [20]. Moreover, the single-layer results for LEM are better than reported results for multi-layer LSTM models, such as in Gal and Ghahramani [14] (2-layer LSTM, 1500 units each: 75.…”

Section: Modelmentioning

confidence: 66%

“…Moreover, the single-layer results for LEM are better than reported results for multi-layer LSTM models, such as in Gal and Ghahramani [14] (2-layer LSTM, 1500 units each: 75. [20] 115.9 256 131k LSTM [20] 116.9 256 524k SkipLSTM [20] 114.2 256 524k TARNN [20] 94.6 256 524k LEM 72.8 256 524k…”

Section: Modelmentioning

confidence: 99%

“…𝜕𝜃 is to observe from the representation formula (20), the bound (24) on matrices F ℓ,ℓ−1 and the order notation (38) that,…”

mentioning

confidence: 99%

See 2 more Smart Citations

Long Expressive Memory for Sequence Modeling

Rusch¹,

Mishra²,

Erichson³

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose a novel method called Long Expressive Memory (LEM) for learning long-term sequential dependencies. LEM is gradient-based, it can efficiently process sequential tasks with very long-term dependencies, and it is sufficiently expressive to be able to learn complicated input-output maps. To derive LEM, we consider a system of multiscale ordinary differential equations, as well as a suitable time-discretization of this system. For LEM, we derive rigorous bounds to show the mitigation of the exploding and vanishing gradients problem, a well-known challenge for gradient-based recurrent sequential learning methods. We also prove that LEM can approximate a large class of dynamical systems to high accuracy. Our empirical results, ranging from image and time-series classification through dynamical systems prediction to speech recognition and language modeling, demonstrate that LEM outperforms state-of-the-art recurrent neural networks, gated recurrent units, and long short-term memory models.

show abstract

Section: Modelmentioning

confidence: 66%

Section: Modelmentioning

confidence: 99%

See 1 more Smart Citation

Long Expressive Memory for Sequence Modeling

Rusch¹,

Mishra²,

Erichson³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Efficient Training for SNNs. Several RNN training methods pursue online learning and constant memory occupation agnostic time horizon, such as real time recurrent learning [60] and forward propagation through time [31]. Inspired by them, some SNN training methods [2,3,[70][71][72] apply similar ideas to achieve memory-efficient and online learning.…”

Section: Related Workmentioning

confidence: 99%

Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks

Meng¹,

Xiao²,

Shen³

et al. 2023

Preprint

View full text Add to dashboard Cite

Spiking Neural Networks (SNNs) are promising energyefficient models for neuromorphic computing. For training the non-differentiable SNN models, the backpropagation through time (BPTT) with surrogate gradients (SG) method has achieved high performance. However, this method suffers from considerable memory cost and training time during training. In this paper, we propose the Spatial Learning Through Time (SLTT) method that can achieve high performance while greatly improving training efficiency compared with BPTT. First, we show that the backpropagation of SNNs through the temporal domain contributes just a little to the final calculated gradients. Thus, we propose to ignore the unimportant routes in the computational graph during backpropagation. The proposed method reduces the number of scalar multiplications and achieves a small memory occupation that is independent of the total time steps. Furthermore, we propose a variant of SLTT, called SLTT-K, that allows backpropagation only at K time steps, then the required number of scalar multiplications is further reduced and is independent of the total time steps. Experiments on both static and neuromorphic datasets demonstrate superior training efficiency and performance of our SLTT. In particular, our method achieves state-of-the-art accuracy on ImageNet, while the memory cost and training time are reduced by more than 70% and 50%, respectively, compared with BPTT.

show abstract

“…This way, they retain major portion of the dynamics and forecast the future behavior of the system. Both, the incremental Recurrent Neural Network (IRNN) (Kag et al, 2019) and the time adaptive RNN (Kag & Saligrama, 2021) use additional recurrent iterations on each input to enable the model of coping different input time scales, where the latter provides a time-varying function that adapts the model's behavior to the time scale of the provided input.…”

Section: Related Workmentioning

confidence: 99%

Flipped Classroom: Effective Teaching for Time Series Forecasting

Teutsch¹,

Mäder²

2022

Preprint

View full text Add to dashboard Cite

Sequence-to-sequence models based on LSTM and GRU are a most popular choice for forecasting time series data reaching state-of-the-art performance. Training such models can be delicate though. The two most common training strategies within this context are teacher forcing (TF) and free running (FR). TF can be used to help the model to converge faster but may provoke an exposure bias issue due to a discrepancy between training and inference phase. FR helps to avoid this but does not necessarily lead to better results, since it tends to make the training slow and unstable instead. Scheduled sampling was the first approach tackling these issues by picking the best from both worlds and combining it into a curriculum learning (CL) strategy. Although scheduled sampling seems to be a convincing alternative to FR and TF, we found that, even if parametrized carefully, scheduled sampling may lead to premature termination of the training when applied for time series forecasting. To mitigate the problems of the above approaches we formalize CL strategies along the training as well as the training iteration scale. We propose several new curricula, and systematically evaluate their performance in two experimental sets. For our experiments, we utilize six datasets generated from prominent chaotic systems. We found that the newly proposed increasing training scale curricula with a probabilistic iteration scale curriculum consistently outperforms previous training strategies yielding an NRMSE improvement of up to 81% over FR or TF training. For some datasets we additionally observe a reduced number of training iterations. We observed that all models trained with the new curricula yield higher prediction stability allowing for longer prediction horizons.

show abstract

Time Adaptive Recurrent Neural Network

Cited by 16 publications

References 15 publications

Long Expressive Memory for Sequence Modeling

Long Expressive Memory for Sequence Modeling

Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks

Flipped Classroom: Effective Teaching for Time Series Forecasting

Contact Info

Product

Resources

About