Video Generation From Single Semantic Label Map

Abstract

This paper proposes the novel task of video generation conditioned on a SINGLE semantic label map, which provides a good balance between flexibility and quality in the generation process. Different from typical end-to-end approaches, which model both scene content and dynamics in a single step, we propose to decompose this difficult task into two sub-problems. As current image generation methods do better than video generation in terms of detail, we synthesize high quality content by only generating the first frame. Then we animate the scene based on its semantic meaning to obtain temporally coherent video, giving us excellent results overall. We employ a cVAE for predicting optical flow as a beneficial intermediate step to generate a video sequence conditioned on the initial single frame. A semantic label map is integrated into the flow prediction module to achieve major improvements in the image-to-video generation process. Extensive experiments on the Cityscapes dataset show that our method outperforms all competing methods. The source code will be released on https://github.com/junting/seg2vid.

Visionnet: A Coarse-To-Fine Motion Prediction Algorithm Based On Active Interaction-Aware Drivable Space Learning

“…Trajectory Prediction We use the fused feature z cat,i of the i-th agent to predict its future trajectories (in multi-modality case). Similar to [6], we employ three deconvolutional layers with stride 2 to upsample z cat and obtain the feature representation r of size R 64×H×W , which can be written as, r = Decode pnet (z cat ) . Another issue is that the future behavior of an agent follows a multimodal distribution.…”

“…All experiments are conducted on a machine with an NVIDIA Tesla V100 GPU. Evaluation Metrics Similar to prior work [6,2,10], two metrics are used to evaluate the compared methods. They are Average Displacement Error (ADE): the mean Euclidean distance between predicted and the realistic trajectory and Final Displacement Error (FDE): the Euclidean distance between the final location of the predicted trajectory and the ground truth.…”

“…Secondly, there are many factors that are relevant to the trajectory of an agent, such as the nature of the ground, the mental state of the driver and the pedestrian, etc. To address these challenges, many trajectory prediction methods have been proposed which can be roughly classified into two categories, i.e., the coordinate-based methods [1,2,3,4] and the vision-based methods [5,6,7,8,9].…”

“…On the other hand, the vision-based methods are able to use semantic images, including camera images, LiDAR point cloud data, and Occupancy Grid Maps (OGMs) [5], for global interaction feature extraction. Work [6] generate the prediction sequences in the form of images by Convolutional Neural Networks (CNNs). Roughly speaking, the vision-based methods demonstrate strong understanding of the physical scene but poor generalization ability to noisy data.…”

Sequence-to-Sequence Video Prediction by Learning Hierarchical Representations

“…PredRNN++ [41] proposed Causal LSTMs with cascaded dual memories to model short-term video. Pan et al [42] proposed to generate video frames from a single semantic map. Wang et al [43] developed a point-to-point network which generates intermediate frames given the start and end frames.…”