Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary

Zou, Nan; Xiang, Zhiyu; Chen, Yiman; Chen, Shuya; Qiao, Chengyu

doi:10.3390/s20030635

Cited by 17 publications

(6 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the context of image processing, tasks such as object detection and semantic segmentation have been tackled successfully (Yao et al, 2012), (Mohan and Valada, 2020), (Kim et al, 2020). Depth estimation and semantic segmentation have also been combined in one CNN as in (Eigen and Fergus, 2014), (Hazirbas et al, 2016), (Kendall et al, 2017), (Zou et al, 2020) Multi task networks have shown to achieve better results as whole in terms of their capability to provide a more holistic representation, but also the performance of each one of the tasks improves as a result of having a multi task CNN. Inspired by this approach, Liebel and Körner (2018) introduced the concept of "auxiliary tasks", they are side tasks that are less relevant for a given application but that potentially improved the performance of the core tasks.…”

Section: Multi Task Leaningmentioning

confidence: 99%

“…Panoptic segmentation, for example, combines instance segmentation, object detection and semantic segmentation (Mohan and Valada, 2020), (Cheng et al, 2020), (Wang et al, 2020), (Weber et al, 2020). To our best knowledge, only few methods have combined semantic segmentation and depth completion (Sanchez-Escobedo et al, 2018), (Zou et al, 2020). As compared to other applications, e.g.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

SemSegDepth: A Combined Model for Semantic Segmentation and Depth Completion

Lagos

Rahtu

2022

Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Application

View full text Add to dashboard Cite

Holistic scene understanding is pivotal for the performance of autonomous machines. In this paper we propose a new end-to-end model for performing semantic segmentation and depth completion jointly. The vast majority of recent approaches have developed semantic segmentation and depth completion as independent tasks. Our approach relies on RGB and sparse depth as inputs to our model and produces a dense depth map and the corresponding semantic segmentation image. It consists of a feature extractor, a depth completion branch, a semantic segmentation branch and a joint branch which further processes semantic and depth information altogether. The experiments done on Virtual KITTI 2 dataset, demonstrate and provide further evidence, that combining both tasks, semantic segmentation and depth completion, in a multi-task network can effectively improve the performance of each task. Code is available at https://github.com/juanb09111/semantic depth.

show abstract

Section: Multi Task Leaningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

SemSegDepth: A Combined Model for Semantic Segmentation and Depth Completion

Lagos

Rahtu

2022

Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Application

View full text Add to dashboard Cite

show abstract

“…RGB-GC [18] uses pre-trained semantic segmentation model on Cityscapes dataset [19]. SSDNet [20] and RSDCN [21] exploit the Virtual KITTI dataset [22] to learn the semantic labels of each pixel in dense depth. DeepLiDAR [1] jointly learns the depth completion and normal prediction tasks where the normal prediction module is pre-trained on a synthetic dataset generated from CARLA simulator [23].…”

Section: Related Workmentioning

confidence: 99%

DenseLiDAR: A Real-Time Pseudo Dense Depth Guided Depth Completion Network

Xiang

et al. 2021

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

Depth Completion can produce a dense depth map from a sparse input and provide a more complete 3D description of the environment. Despite great progress made in depth completion, the sparsity of the input and low density of the ground truth still make this problem challenging. In this work, we propose DenseLiDAR, a novel real-time pseudodepth guided depth completion neural network. We exploit dense pseudo-depth map obtained from simple morphological operations to guide the network in three aspects: (1) Constructing a residual structure for the output; (2) Rectifying the sparse input data; (3) Providing dense structural loss for training the network. Thanks to these novel designs, higher performance of the output could be achieved. In addition, two new metrics for better evaluating the quality of the predicted depth map are also presented. Extensive experiments on KITTI depth completion benchmark suggest that our model is able to achieve the state-of-the-art performance at the highest frame rate of 50Hz. The predicted dense depth is further evaluated by several downstream robotic perception or positioning tasks. For the task of 3D object detection, 3˜5 percent performance gains on small objects categories are achieved on KITTI 3D object detection dataset. For RGB-D SLAM, higher accuracy on vehicle's trajectory is also obtained in KITTI Odometry dataset. These promising results not only verify the high quality of our depth prediction, but also demonstrate the potential of improving the related downstream tasks by using depth completion results.

show abstract

“…This paper works on cityscape with depth dataset, which has many weakly labeled images. The challenge with working with 3D or 2.5D datasets is that they are not many accurate and highresolution datasets in that field [2,3].…”

Section: Introductionmentioning

confidence: 99%

Street Scene understanding via Semantic Segmentation Using Deep Learning

Noori¹,

Shaker²,

Azeez³

2022

ETJ

View full text Add to dashboard Cite

 Scene classification is an essential conception task used by robotics for understanding the environment.  The deep learning technique has been proved as a great role in the challenging scene understanding application.  Using data augmentation to increase dataset size  Using K-means clustering as a preprocessor for the input dataset  The proposed hydride model is generated by combined two of the deep , deep neural networks as an xception and U-net models.Scene classification is an essential conception task used by robotics for understanding the environment. Like the street scene, the outdoor scene is composed of images with depth that has a greater variety than iconic object images. Image semantic segmentation is an important task for Autonomous driving and M obile robotics applications because it introduces enormous information needed for safe navigation and complex reasoning. This paper provides a model for semantic segmentation of outdoor sense to classify each object in the scene. The proposed network model generates a hybrid model that combines U-NET with Xception networks to work on 2.5 dimensions cityscape dataset, which is used for 3D applications. This process contains two stages. The first is the pre-processing operation on the RGB-D dataset (data Augmentation and k-means cluster). The second stage designed the hybrid model, which achieves a pixel accuracy is 0.7874. The output module is generated using a computer with GPU memory NVIDIA GeForce RTX 2060 6G, programming with python 3.7.

show abstract

Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary

Cited by 17 publications

References 35 publications

SemSegDepth: A Combined Model for Semantic Segmentation and Depth Completion

SemSegDepth: A Combined Model for Semantic Segmentation and Depth Completion

DenseLiDAR: A Real-Time Pseudo Dense Depth Guided Depth Completion Network

Street Scene understanding via Semantic Segmentation Using Deep Learning

Contact Info

Product

Resources

About