MS-RNN: A Flexible Multi-Scale Framework for Spatiotemporal Predictive Learning

Abstract

Spatiotemporal predictive learning, which predicts future frames through historical prior knowledge with the aid of deep learning, is widely used in many fields. Previous work essentially improves the model performance by widening or deepening the network, but it also brings surging memory overhead, which seriously hinders the development and application of this technology. In order to improve the performance without increasing memory consumption, we focus on scale, which is another dimension to improve model performance but with low memory requirement. The effectiveness has been widely proved in many CNN-based tasks such as image classification and semantic segmentation, but it has not been fully explored in recent RNN models. In this paper, learning from the benefit of multi-scale, we propose a general framework named Multi-Scale RNN (MS-RNN) to boost recent RNN models for spatiotemporal predictive learning. By integrating different scales, we enhance the existing models with both improved performance and greatly reduced overhead. We verify our MS-RNN framework by exhaustive experiments with 6 popular RNN models (ConvLSTM, TrajGRU, PredRNN, PredRNN++, MIM, and MotionRNN) on 4 different datasets (Moving MNIST, KTH, TaxiBJ, and HKO-7). The results show the efficiency that the RNN models incorporating our framework have much lower memory cost but better performance than before. Our code is released at https://github.com/mazhf/MS-RNN.

DB-RNN: An RNN for Precipitation Nowcasting Deblurring

“…Their origin is the ConvLSTM [16] proposed by Shi et al in 2015. Later, a large number of followers continue to refresh the forecasting performance, for example, Traj-GRU [8], PredRNN [28], PredRNN++ [45], MIM [29], Mo-tionRNN [31], PrecipLSTM [30], MS-RNN [32], and MS-LSTM [26]. GAN [17], [19], [46]- [48] uses UNet or Con-vRNN as the generator and uses one or more discriminators with different roles to play the minimax game.…”

“…Although they gain stronger forecasting capabilities, they will consume huge memory and computing resources. Multiscale RNN (MS-RNN [32]) proposes to adopt a multiscale architecture to improve these ConvRNN models, which will make them have less memory and computing requirements but stronger spatiotemporal modeling capabilities while keeping the number of parameters constant. Specifically, MS-RNN reintegrates these increasingly complex tensor flows and embeds the UNet [40] IV.…”

“…Nevertheless, the earth system is chaotic, high dimensional, and spatiotemporal [27], and it is a particularly challenging task to obtain a robust precipitation representation system. Since introducing a reasonable inductive bias [1] (prior assumption) about the domain knowledge when designing a neural network will reduce the training samples and training difficulty, models [8], [16], [28]- [32] integrating convolutional neural network (CNN) and recurrent neural network (RNN) stand out among many competitors and is enduring. We call this type of model ConvRNN, which obeys the Markov assumption like RNN and can simultaneously model spatiotemporal patterns.…”

“…In this paper, we propose the deblurring RNN (DB-RNN) to alleviate the autoregression symptom, which consists of two cascaded ConvRNN networks, one for forecasting and one for deblurring. For efficiency, we use a more advanced multiscale version of ConvRNN called MS-RNN [32], which is compatible with many ConvRNNs. We train the two parts sequentially by controlling the corresponding loss weights and finally obtain a powerful end-to-end network for precipitation nowcasting.…”

Focal Frame Loss: A Simple but Effective Loss for Precipitation Nowcasting

“…The RNN models are dominated by ConvLSTM [16] and its variants, such as Traj-GRU [17], PredRNN [18], PredRNN++ [22], E3D-LSTM [23], MIM [19], CubicLSTM [37], SA-ConvLSTM [24], Motion-RNN [20], and so on. These RNN models are getting wider and deeper [38]. Although they alleviate the prediction ambiguity problem to some extent, it also brings a significant increase in computational cost.…”