Sequence-to-Sequence Video Prediction by Learning Hierarchical Representations

Abstract

Video prediction which maps a sequence of past video frames into realistic future video frames is a challenging task because it is difficult to generate realistic frames and model the coherent relationship between consecutive video frames. In this paper, we propose a hierarchical sequence-to-sequence prediction approach to address this challenge. We present an end-to-end trainable architecture in which the frame generator automatically encodes input frames into different levels of latent Convolutional Neural Network (CNN) features, and then recursively generates future frames conditioned on the estimated hierarchical CNN features and previous prediction. Our design is intended to automatically learn hierarchical representations of video and their temporal dynamics. Convolutional Long Short-Term Memory (ConvLSTM) is used in combination with skip connections so as to separately capture the sequential structures of multiple levels of hierarchy of features. We adopt Scheduled Sampling for training our recurrent network in order to facilitate convergence and to produce high-quality sequence predictions. We evaluate our method on the Bouncing Balls, Moving MNIST, and KTH human action dataset, and report favorable results as compared to existing methods.

Next frame prediction using ConvLSTM

“…Authors in [9] explored a topic, namely whether network topologies are necessary, and instead propose a new approach: the main goal is to find the video prediction algorithm with the least inductive bias while optimising network capacity. Finally, it is shown that by just boosting the capacity of a regular neural network, a high-quality video prediction may be produced even without the usage of the previously stated techniques (such as adversarial objectives, optical flows, and so on) [9].…”

Unsupervised Learning of Disentangled Representation via Auto-Encoding: A Survey

“…Further, they can be used to generate novel examples not found in the original dataset [ 21 ]. Such feature learning also supports a variety of other applications, such as super-resolution [ 22 ], multimodal application [ 23 , 24 , 25 , 26 , 27 ], medical imaging [ 28 , 29 ], video prediction [ 30 , 31 , 32 , 33 , 34 ], natural language processing [ 35 , 36 , 37 ], transfer learning and zero-shot learning [ 38 ].…”