Video Imagination from a Single Image with Transformation Generation

Chen, Baoyang; Wang, Wenmin; Wang, Jinzhuo

doi:10.1145/3126686.3126737

Cited by 44 publications

(28 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…All aforementioned work focuses on how to directly predict future frames. Different from these work, recently, people propose to predict transformations needed for generating future frames [33] and [4], which further boosts the performance of video prediction.…”

Section: Video Frame Predictionmentioning

confidence: 99%

Future Frame Prediction for Anomaly Detection - A New Baseline

Luo

Lian

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

1,119

1,029

View full text Add to dashboard Cite

Anomaly detection in videos refers to the identification of events that do not conform to expected behavior. However, almost all existing methods tackle the problem by minimizing the reconstruction errors of training data, which cannot guarantee a larger reconstruction error for an abnormal event. In this paper, we propose to tackle the anomaly detection problem within a video prediction framework. To the best of our knowledge, this is the first work that leverages the difference between a predicted future frame and its ground truth to detect an abnormal event. To predict a future frame with higher quality for normal events, other than the commonly used appearance (spatial) constraints on intensity and gradient, we also introduce a motion (temporal) constraint in video prediction by enforcing the optical flow between predicted frames and ground truth frames to be consistent, and this is the first work that introduces a temporal constraint into the video prediction task. Such spatial and motion constraints facilitate the future frame prediction for normal events, and consequently facilitate to identify those abnormal events that do not conform the expectation. Extensive experiments on both a toy dataset and some publicly available datasets validate the effectiveness of our method in terms of robustness to the uncertainty in normal events and the sensitivity to abnormal events. All codes are released in https://github. com/StevenLiuWen/ano_pred_cvpr2018.We also compare the video prediction network based and Auto-Encoder network based anomaly detection. Here for Auto-Encoder network based anomaly detection, we use the Conv-AE [13] which is the latest work and achieves state-of-the-art performance for anomaly detection. Because of the capacity of deep neural network, Auto-Encoder

show abstract

Section: Video Frame Predictionmentioning

confidence: 99%

Future Frame Prediction for Anomaly Detection - A New Baseline

Luo

Lian

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

1,119

1,029

View full text Add to dashboard Cite

show abstract

“…As a result, recent research focused on Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) for image prediction, and generation problems. 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 …”

Section: Methodsmentioning

confidence: 99%

“…Due to the ambiguous nature of dynamic frames prediction, stochastic models outperform deterministic models, and loss functions. As a result, recent research focused on Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) for image prediction, and generation problems [32][33][34][35][36][37][38][39][40] .…”

Section: Learning Modelmentioning

confidence: 99%

Fast dynamic brain PET imaging using stochastic variational prediction for recurrent frame generation

et al. 2021

View full text Add to dashboard Cite

We assess the performance of a recurrent frame generation algorithm for prediction of late frames from initial frames in dynamic brain PET imaging.Methods: Clinical dynamic 18 F-DOPA brain PET/CT studies of 46 subjects with ten folds crossvalidation were retrospectively employed. A novel stochastic adversarial video prediction model was implemented to predict the last 13 frames (25-90 min) from the initial 13 frames (0-25 min). The quantitative analysis of the predicted dynamic PET frames was performed for the test and validation dataset using established metrics. Results:The predicted dynamic images demonstrated that the model is capable of predicting the trend of change in time-varying tracer biodistribution. The Bland-Altman plots reported the lowest tracer uptake bias (-0.04) for the putamen region and the smallest variance (95% CI: -0.38, +0.14) for the cerebellum. The region-wise Patlak graphical analysis in the caudate and putamen regions for 8 subjects from the test and validation dataset showed that the average bias for and distribution volume was 4.3%, 5.1% and 4.4%, 4.2%, (p-value < 0.05), respectively. Conclusion:We have developed a novel deep learning approach for fast dynamic brain PET imaging capable of generating the last 65 min time frames from the initial 25 min frames, thus enabling significant reduction in scanning time.

show abstract

“…In [4], the authors predict or imagine multiple frames from a single image. They generate affine transformations between each frame, and apply them to the original input image to produce their prediction.…”

Section: Non-human Motion Predictionmentioning

confidence: 99%

HP-GAN: Probabilistic 3D Human Motion Prediction via GAN

Barsoum

Kender

Liu

2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

287

192

View full text Add to dashboard Cite

Predicting and understanding human motion dynamics has many applications, such as motion synthesis, augmented reality, security, and autonomous vehicles. Due to the recent success of generative adversarial networks (GAN), there has been much interest in probabilistic estimation and synthetic data generation using deep neural network architectures and learning algorithms.We propose a novel sequence-to-sequence model for probabilistic human motion prediction, trained with a modified version of improved Wasserstein generative adversarial networks (WGAN-GP), in which we use a custom loss function designed for human motion prediction. Our model, which we call HP-GAN, learns a probability density function of future human poses conditioned on previous poses. It predicts multiple sequences of possible future human poses, each from the same input sequence but a different vector z drawn from a random distribution. Furthermore, to quantify the quality of the non-deterministic predictions, we simultaneously train a motion-quality-assessment model that learns the probability that a given skeleton sequence is a real human motion.We test our algorithm on two of the largest skeleton datasets: NTURGB-D and Human3.6M. We train our model on both single and multiple action types. Its predictive power for long-term motion estimation is demonstrated by generating multiple plausible futures of more than 30 frames from just 10 frames of input. We show that most sequences generated from the same input have more than 50% probabilities of being judged as a real human sequence. We will release all the code used in this paper to Github.

show abstract

Video Imagination from a Single Image with Transformation Generation

Cited by 44 publications

References 35 publications

Future Frame Prediction for Anomaly Detection - A New Baseline

Future Frame Prediction for Anomaly Detection - A New Baseline

Fast dynamic brain PET imaging using stochastic variational prediction for recurrent frame generation

HP-GAN: Probabilistic 3D Human Motion Prediction via GAN

Contact Info

Product

Resources

About