A Review on Deep Learning Techniques for Video Prediction

Abstract

The ability to predict, anticipate and reason about future outcomes is a key component of intelligent decision-making systems. In light of the success of deep learning in computer vision, deep-learning-based video prediction emerged as a promising research direction. Defined as a self-supervised learning task, video prediction represents a suitable framework for representation learning, as it demonstrated potential capabilities for extracting meaningful representations of the underlying patterns in natural videos. Motivated by the increasing interest in this task, we provide a review on the deep learning methods for prediction in video sequences. We firstly define the video prediction fundamentals, as well as mandatory background concepts and the most used datasets. Next, we carefully analyze existing video prediction models organized according to a proposed taxonomy, highlighting their contributions and their significance in the field. The summary of the datasets and methods is accompanied with experimental results that facilitate the assessment of the state of the art on a quantitative basis. The paper is summarized by drawing some general conclusions, identifying open research challenges and by pointing out future research directions.

Enlargement of the Field of View Based on Image Region Prediction Using Thermal Videos

“…Image prediction was performed in [30] using a visible light image and a stochastic variational video prediction (SV2P) method. In a review study [31], the datasets from 2004 to 2019 used in image prediction were compared with the image prediction models that were released between 2014 and 2020. In the survey in [32], studies on and datasets for image prediction were explained.…”

DFPN: Deformable Frame Prediction Network

“…Comprehensive reviews on video frame prediction methods can be found in [7,8]. Methods covered in these reviews can be classified according to their network architecture, prediction methodology, and loss function used in training.…”

A Novel Multi-Input Multi-Output Recurrent Neural Network Based on Multimodal Fusion and Spatiotemporal Prediction for 0–4 Hour Precipitation Nowcasting

“…This approach uses the generated optical flow [20] to guide the connection structure in the network, and the point in the convolution structure is connected to points with a higher correlation instead of a fixed number of surrounding points. Many spatiotemporal prediction methods [21][22][23] have regarded radar echo extrapolation as one of the tasks to evaluate the spatiotemporal prediction ability of their methods. Wang et al [24] proposed a spatiotemporal prediction method called PredRNN, which makes the spatial features of each layer of ConvLSTM interact in time series.…”