Improving Semantic Segmentation via Video Propagation and Label Relaxation

Abstract

Semantic segmentation requires large amounts of pixelwise annotations to learn accurate models. In this paper, we present a video prediction-based methodology to scale up training sets by synthesizing new training samples in order to improve the accuracy of semantic segmentation networks. We exploit video prediction models' ability to predict future frames in order to also predict future labels. A joint propagation strategy is also proposed to alleviate mis-alignments in synthesized samples. We demonstrate that training segmentation models on datasets augmented by the synthesized samples leads to significant improvements in accuracy. Furthermore, we introduce a novel boundary label relaxation technique that makes training robust to annotation noise and propagation artifacts along object boundaries. Our proposed methods achieve state-of-the-art mIoUs of 83.5% on Cityscapes and 82.9% on CamVid. Our single model, without model ensembles, achieves 72.8% mIoU on the KITTI semantic segmentation test set, which surpasses the winning entry of the ROB challenge 2018. Our code and videos can be found at https://nv-adlr.github. io/publication/2018-Segmentation.

Boosting Real-Time Driving Scene Parsing With Shared Semantics

“…Xu et al [16] applied different segmentation strategies to various regions of the input image, which exploited optical flow to preserve the semantics in static regions. Zhu et al [17] investigated the generation of future semantic segmentation labels from current manual labels by video prediction based on motion vector estimation.…”

The Importance of Loss Functions for Increasing the Generalization Abilities of a Deep Learning-Based Next Frame Prediction Model for Traffic Scenes

“…A second approach to handle the problem of implausible prediction outputs that lack realism is to reduce the complexity of the problem. Many authors, for example, used data with lower-dimensional image content, such as label images, instead of natural image scenes [12,15,[22][23][24][25]. Others split the problem into two problems, motion and content prediction, and learn separate representations for the static and dynamic components.…”

Concept on Landmark Detection in Road Scene Images Taken From a Top-View Camera System

“…With an evaluation time of one second, this algorithm is too slow for robotic applications or automated driving. (Zhu et al, 2019) show a video-based approach to further improve the segmentation process by propagating labels between two frames jointly. In contrast to that, (Chen et al, 2018) show a network capable of close to real-time semantic segmentation.…”