Depth Estimation and Semantic Segmentation from a Single RGB Image Using a Hybrid Convolutional Neural Network

Abstract

Semantic segmentation and depth estimation are two important tasks in computer vision, and many methods have been developed to tackle them. Commonly these two tasks are addressed independently, but recently the idea of merging these two problems into a sole framework has been studied under the assumption that integrating two highly correlated tasks may benefit each other to improve the estimation accuracy. In this paper, depth estimation and semantic segmentation are jointly addressed using a single RGB input image under a unified convolutional neural network. We analyze two different architectures to evaluate which features are more relevant when shared by the two tasks and which features should be kept separated to achieve a mutual improvement. Likewise, our approaches are evaluated under two different scenarios designed to review our results versus single-task and multi-task methods. Qualitative and quantitative experiments demonstrate that the performance of our methodology outperforms the state of the art on single-task approaches, while obtaining competitive results compared with other multi-task methods.

Latent 3D Volume for Joint Depth Estimation and Semantic Segmentation from a Single Image

“…The thresholded accuracy means the ratio of the maximum relative error below the threshold . In our experiments, we use to align the setting with previous studies [ 6 , 8 , 13 , 15 , 16 , 17 , 42 ].…”

“…Multi-task learning with the goal of learning multiple tasks simultaneously [ 40 ] is nowadays based on CNNs [ 13 , 15 , 16 , 17 , 18 , 19 , 20 , 41 , 42 , 43 , 44 , 44 ]. Eigen and Fergus proposed a network structure that can estimate the depth, semantic labels, and the surface orientation of a scene [ 13 ].…”

“…They also proposed an attention-driven loss to deal with long-tail data distribution. Lin et al [ 17 ] presented a hybrid CNN that combines a common feature extraction network and a global depth estimation network for understanding the global layout of the scene. Nekrasov et al [ 16 ] used knowledge distillation and achieved a real-time performance.…”

“…However, these 2D representations are limited to obtain the 2D contextual information in an image due to 2D CNNs. In contrast to existing approaches [ 15 , 16 , 17 ], we propose a 3D feature representation, namely, a latent 3D volume. This representation enhances the 3D contextual information by introducing 3D CNNs.…”

Multi-Scale Spatio-Temporal Feature Extraction and Depth Estimation from Sequences by Ordinal Classification

“…Monocular Depth Estimation [22][23][24][25][26][27][28] infers depth information from single RGB images and is demonstrated to be an ill-posed problem(In most cases, there are several possible outputs corresponding to a given input image and the problem can be seen as a task of selecting the most proper one from all the possible outputs [29]). Stereo Depth Estimation [30][31][32] needs specific devices to capture stereo images and can provide much more clues to estimate the depth than Monocular Depth Estimation.…”