Mixpred: Video Prediction Beyond Optical Flow

Yan, Jie; Qin, Guanshi; Zhao, Rui; Liang, Yan; Xu, Qiang

doi:10.1109/access.2019.2961383

Cited by 4 publications

(4 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Convolutional networks, considered as feed-forward neural networks, are also commonly used in future prediction problems. A multi-model is defined in [28] to model dynamic patterns and learn image representation by combining temporal and spatial sub-networks. In [29], Deep Voxel Flow (DVF) is trained to synthesize future frames by flowing pixel values directly from input frames.…”

Section: Related Workmentioning

confidence: 99%

A Novel Video Prediction Algorithm Based on Robust Spatiotemporal Convolutional Long Short-Term Memory (Robust-ST-ConvLSTM)

Saideni

Helbert

Courreges

et al. 2022

Proceedings of Seventh International Congress on Information and Communication Technology

View full text Add to dashboard Cite

Recently, video prediction algorithms based on neural networks have become a promising research direction. Therefore, a new recurrent video prediction algorithm called "Robust Spatiotemporal Convolutional Long Short-Term Memory" (Robust-ST-ConvLSTM) is proposed in this paper. Robust-ST-ConvLSTM proposes a new internal mechanism that is able to regulate efficiently the flow of spatiotemporal information from video signals based on higher order Convolutional-LSTM. The spatiotemporal information is carried through the entire network to optimize and control the prediction potential of the ConvLSTM cell. In addition, in traditional ConvLSTM units, cell states, that carry relevant information throughout the processing of the input sequence, are updated using only one previous hidden state, which holds information on previous data unit already seen by the network. However, our Robust-ST-ConvLSTM unit will rely on N previous hidden states, that provide temporal context for the motion in video scenes, in the cell state updating process. Experimental results further suggest that the proposed architecture can improve the state-of-the-art video prediction methods significantly on two challenging datasets, including the standard Moving MNIST dataset, and the commonly used video prediction KTH dataset, as human motion dataset.

show abstract

Section: Related Workmentioning

confidence: 99%

A Novel Video Prediction Algorithm Based on Robust Spatiotemporal Convolutional Long Short-Term Memory (Robust-ST-ConvLSTM)

Saideni

Helbert

Courreges

et al. 2022

Proceedings of Seventh International Congress on Information and Communication Technology

View full text Add to dashboard Cite

show abstract

“…This method shows the effectiveness of modelling coherent spatio-temporal fields. Mixpred neural network has been proposed to model the dynamic pattern and learn appearance representations based on given video frames in [ 25 ]. A 3D CNN is utilized into RNN in [ 26 ], which extends representations in temporal dimension and makes the memory unit store better long-term representations.…”

Section: Literature Reviewmentioning

confidence: 99%

“…A 3D CNN is utilized into RNN in [ 26 ], which extends representations in temporal dimension and makes the memory unit store better long-term representations. However, convolutional operations [ 24 , 25 , 26 ] account for short-range intraframe dependencies due to their limited receptive fields and the lack of explicit inter-frame modeling capabilities. The generative adversarial networks [ 8 ] is another approach for spatiotemporal prediction.…”

Section: Literature Reviewmentioning

confidence: 99%

Dual Memory LSTM with Dual Attention Neural Network for Spatiotemporal Prediction

Guan

2021

Sensors

View full text Add to dashboard Cite

Spatiotemporal prediction is challenging due to extracting representations being inefficient and the lack of rich contextual dependences. A novel approach is proposed for spatiotemporal prediction using a dual memory LSTM with dual attention neural network (DMANet). A new dual memory LSTM (DMLSTM) unit is proposed to extract the representations by leveraging differencing operations between the consecutive images and adopting dual memory transition mechanism. To make full use of historical representations, a dual attention mechanism is designed to capture long-term spatiotemporal dependences by computing the correlations between the current hidden representations and the historical hidden representations from temporal and spatial dimensions, respectively. Then, the dual attention is embedded into DMLSTM unit to construct a DMANet, which enables the model with greater modeling power for short-term dynamics and long-term contextual representations. An apparent resistivity map (AR Map) dataset is proposed in this paper. The B-spline interpolation method is utilized to enhance AR Map dataset and makes apparent resistivity trend curve continuous derivative in the time dimension. The experimental results demonstrate that the developed method has excellent prediction performance by comparisons with some state-of-the-art methods.

show abstract

“…The other approach in future frame prediction is a multi-model composed of the above-mentioned approaches. Yan et al [26] studied future frame prediction with a multi-model composed of the temporal and spatial sub-network. Kim et al [9] detected key-points in input frames, and translated the frame according to key-points motion.…”

Section: Related Workmentioning

confidence: 99%

Adaptive Future Frame Prediction with Ensemble Network

Kim

Tanaka

Okutomi

et al. 2020

Preprint

View full text Add to dashboard Cite

Future frame prediction in videos is a challenging problem because videos include complicated movements and large appearance changes. Learning-based future frame prediction approaches have been proposed in kinds of literature. A common limitation of the existing learning-based approaches is a mismatch of training data and test data. In the future frame prediction task, we can obtain the ground truth data by just waiting for a few frames. It means we can update the prediction model online in the test phase. Then, we propose an adaptive update framework for the future frame prediction task. The proposed adaptive updating framework consists of a pre-trained prediction network, a continuous-updating prediction network, and a weight estimation network. We also show that our pre-trained prediction model achieves comparable performance to the existing state-of-the-art approaches. We demonstrate that our approach outperforms existing methods especially for dynamically changing scenes.

show abstract

Mixpred: Video Prediction Beyond Optical Flow

Cited by 4 publications

References 22 publications

A Novel Video Prediction Algorithm Based on Robust Spatiotemporal Convolutional Long Short-Term Memory (Robust-ST-ConvLSTM)

A Novel Video Prediction Algorithm Based on Robust Spatiotemporal Convolutional Long Short-Term Memory (Robust-ST-ConvLSTM)

Dual Memory LSTM with Dual Attention Neural Network for Spatiotemporal Prediction

Adaptive Future Frame Prediction with Ensemble Network

Contact Info

Product

Resources

About