SDNet: Semantically Guided Depth Estimation Network

Ochs, Matthias; Kretz, Adrian; Mester, Rudolf

doi:10.48550/arxiv.1907.10659

Cited by 3 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our approach also outperforms other methods that leverage semantic information by a substantial margin, even those using ground-truth KITTI semantic segmentation and depth labels during train- ing (Ochs et al, 2019). Furthermore, in Figure 5 we also present qualitative results showing the improvements in depth estimation generated by our proposed framework, compared to our baseline.…”

Section: Depth Estimation Performancementioning

confidence: 82%

“…The second category attempts to learn both tasks in a single framework, and uses consistency losses to ensure that both are optimized simultaneously and regularize each other, so the information contained in one task can be transferred to improve the other. For instance, Ochs et al (2019) estimated depth with an ordinal classification loss similar to the standard semantic classification loss, and used empirical weighting to combine them into a single loss for optimization. Similarly, Chen et al (2019) used a unified conditional decoder that can generate either semantic or depth estimates, and both outputs are used to generate a series of losses also combined using empirical weighting to generate the final loss to be optimized.…”

Section: Related Workmentioning

confidence: 99%

“…For instance, a car 25 meters away appears smaller (on the image plane) than a car only 5 meters away but bigger than a truck 50 meters away. Current depth estimation methods either do not leverage this structure explicitly or rely on strong semantic supervision to jointly optimize geometric consistency and a semantic proxy task in a multi-task objective (Ochs et al, 2019;Chen et al, 2019), thus departing from the self-supervised paradigm.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Semantically-Guided Representation Learning for Self-Supervised Monocular Depth

Guizilini¹,

Hou²,

Li³

et al. 2020

Preprint

View full text Add to dashboard Cite

Self-supervised learning is showing great promise for monocular depth estimation, using geometry as the only source of supervision. Depth networks are indeed capable of learning representations that relate visual appearance to 3D properties by implicitly leveraging category-level patterns. In this work we investigate how to leverage more directly this semantic structure to guide geometric representation learning, while remaining in the self-supervised regime. Instead of using semantic labels and proxy losses in a multi-task approach, we propose a new architecture leveraging fixed pretrained semantic segmentation networks to guide self-supervised representation learning via pixel-adaptive convolutions. Furthermore, we propose a two-stage training process to overcome a common semantic bias on dynamic objects via resampling. Our method improves upon the state of the art for self-supervised monocular depth prediction over all pixels, fine-grained details, and per semantic categories. †

show abstract

Section: Depth Estimation Performancementioning

confidence: 82%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Semantically-Guided Representation Learning for Self-Supervised Monocular Depth

Guizilini¹,

Hou²,

Li³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…In addressing the aforementioned challenge, one of the prevailing methodologies involves the application of Convolutional Neural Networks (CNNs). This widely adopted approach is extensively discussed in [ 15 ], wherein networks like SANET [ 16 ] and SDNET [ 17 ] are employed. SANET excels at automatically discerning spatial attributes such as color, while SDNET specializes in extracting shape-related features, like facial contours.…”

Section: Related Workmentioning

confidence: 99%

A Comprehensive Study on Pain Assessment from Multimodal Sensor Data

Benavent-Lledo,

Mulero-Pérez,

Ortiz-Perez

et al. 2023

Sensors

View full text Add to dashboard Cite

Pain assessment is a critical aspect of healthcare, influencing timely interventions and patient well-being. Traditional pain evaluation methods often rely on subjective patient reports, leading to inaccuracies and disparities in treatment, especially for patients who present difficulties to communicate due to cognitive impairments. Our contributions are three-fold. Firstly, we analyze the correlations of the data extracted from biomedical sensors. Then, we use state-of-the-art computer vision techniques to analyze videos focusing on the facial expressions of the patients, both per-frame and using the temporal context. We compare them and provide a baseline for pain assessment methods using two popular benchmarks: UNBC-McMaster Shoulder Pain Expression Archive Database and BioVid Heat Pain Database. We achieved an accuracy of over 96% and over 94% for the F1 Score, recall and precision metrics in pain estimation using single frames with the UNBC-McMaster dataset, employing state-of-the-art computer vision techniques such as Transformer-based architectures for vision tasks. In addition, from the conclusions drawn from the study, future lines of work in this area are discussed.

show abstract

“…Joint Depth and Semantics Modeling. Several works [6,36,44,45] model the depth estimation and semantic (panoptic) segmentation as a multi-task learning procedure and bond them with a series of consistent losses. Some other works [17,28,54] utilize semantic segmentation to guide depth estimation learning.…”

Section: Related Workmentioning

confidence: 99%

PolyphonicFormer: Unified Query Learning for Depth-aware Video Panoptic Segmentation

Yuan,

Li,

Yang

et al. 2021

Preprint

View full text Add to dashboard Cite

The recently proposed Depth-aware Video Panoptic Segmentation (DVPS) aims to predict panoptic segmentation results and depth maps in a video, which is a challenging scene understanding problem. In this paper, we present PolyphonicFormer, a vision transformer to unify all the subtasks under the DVPS task. Our method explores the relationship between depth estimation and panoptic segmentation via query-based learning. In particular, we design three different queries including thing query, stuff query, and depth query. Then we propose to learn the correlations among these queries via gated fusion. From the experiments, we prove the benefits of our design from both depth estimation and panoptic segmentation aspects. Since each thing query also encodes the instance-wise information, it is natural to perform tracking via cropping instance mask features with appearance learning. Our method ranks 1st on the ICCV-2021 BMTT Challenge video + depth track. Ablation studies are reported to show how we improve the performance. Code will be available here.

show abstract

SDNet: Semantically Guided Depth Estimation Network

Cited by 3 publications

References 0 publications

Semantically-Guided Representation Learning for Self-Supervised Monocular Depth

Semantically-Guided Representation Learning for Self-Supervised Monocular Depth

A Comprehensive Study on Pain Assessment from Multimodal Sensor Data

PolyphonicFormer: Unified Query Learning for Depth-aware Video Panoptic Segmentation

Contact Info

Product

Resources

About