DensePASS: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation with Attention-Augmented Context Exchange

Ma, Chaoxiang; Zhang, Jiaming; Yang, Kailun; Roitberg, Alina; Stiefelhagen, Rainer

doi:10.1109/itsc48978.2021.9564920

Cited by 27 publications

(24 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recently, to address the dearth of annotated panoramic images, researchers have explicitly formalized panoramic segmentation as an unsupervised domain adaptation problem or a domain generalization problem by transferring from the data-rich pinhole domain to the data-scarce panoramic domain [253], [255]. For domain adaptation, one can use labeled pinhole data as the source domain and unlabeled panoramic images as the target domain.…”

Section: A Semantic Scene Understanding With Image Segmentationmentioning

confidence: 99%

“…For domain generalization, one can only use the source domain images, with the aim to produce a robust, generalized segmentation model in the target domain. P2PDA [253], [254] (Fig. 11(c)) designed attentionaugmented domain adaptation modules to detect and magnify the pinhole-panoramic correspondences in multiple spaces.…”

Section: A Semantic Scene Understanding With Image Segmentationmentioning

confidence: 99%

See 1 more Smart Citation

Review on Panoramic Imaging and Its Applications in Scene Understanding

Gao¹,

Yang²,

Sheng³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

With the rapid development of high-speed communication and artificial intelligence technologies, human perception of real-world scenes is no longer limited to the use of small Field of View (FoV) and low-dimensional scene detection devices. Panoramic imaging emerges as the next generation of innovative intelligent instruments for environmental perception and measurement. However, while satisfying the need for large-FoV photographic imaging, panoramic imaging instruments are expected to have high resolution, no blind area, miniaturization, and multi-dimensional intelligent perception, and can be combined with artificial intelligence methods towards the next generation of intelligent instruments, enabling deeper understanding and more holistic perception of 360 • real-world surrounding environments. Fortunately, recent advances in freeform surfaces, thinplate optics, and metasurfaces provide innovative approaches to address human perception of the environment, offering promising ideas beyond conventional optical imaging. In this review, we begin with introducing the basic principles of panoramic imaging systems, and then describe the architectures, features, and functions of various panoramic imaging systems. Afterwards, we discuss in detail the broad application prospects and great design potential of freeform surfaces, thin-plate optics, and metasurfaces in panoramic imaging. We then provide a detailed analysis on how these techniques can help enhance the performance of panoramic imaging systems. We further offer a detailed analysis of applications of panoramic imaging in scene understanding for autonomous driving and robotics, spanning panoramic semantic image segmentation, panoramic depth estimation, panoramic visual localization, and so on. Finally, we cast a perspective on future potential and research directions for panoramic imaging instruments.

show abstract

Section: A Semantic Scene Understanding With Image Segmentationmentioning

confidence: 99%

Section: A Semantic Scene Understanding With Image Segmentationmentioning

confidence: 99%

Review on Panoramic Imaging and Its Applications in Scene Understanding

Gao¹,

Yang²,

Sheng³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Deep learning-based approaches. Since the rise of deep learning, approaches based on Convolutional Neural Networks (CNNs), which operate directly on the image input and learn intermediate representations end-to-end, took over the top of most computer vision benchmarks, ranging from object classification and -detection to action recognition and semantic segmentation [49]- [55]. These developments also had strong influences on the field of driver observation where a variety of works report top performance of CNN-based architectures [1], [5], [13], [37], [56]- [62], with spatiotemporal CNNs, such as I3D [52], P3D [51], and 3D-MobileNet [63] being popular backbones [1], [13], [60], [61], [64].…”

Section: Related Work a Driver Action Recognitionmentioning

confidence: 99%

Is my Driver Observation Model Overconfident? Input-guided Calibration Networks for Reliable and Interpretable Confidence Estimates

Roitberg¹,

Peng²,

Yang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

“…P ANORAMIC semantic segmentation offers an omnidirectional and dense visual understanding regimen that integrates 360 • perception of surrounding scenes and pixel-wise predictions of input images [1]. The attracted attention of 360 • cameras are manifesting, with an increasing number of learning systems and practical applications, such as holistic sensing in autonomous vehicles [2], [3], [4] and immersive viewing in augmented-and virtual reality (AR/VR) devices [5], [6], [7]. In contrast to images captured via pinhole cameras (Fig.…”

Section: Introductionmentioning

confidence: 99%

“…Based on the MPA strategy, we first revisit the Pinhole-to-Panoramic (PIN2PAN) paradigm as previous works [3], [9], by considering the label-rich pinhole images as the source domain and the label-scare panoramic images as the target domain. Furthermore, a new dataset (SynPASS) with 9,080 synthetic panoramic images is created by using the CARLA simulator [18].…”

Section: Introductionmentioning

confidence: 99%

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

Zhang¹,

Yang²,

Sheng³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we address panoramic semantic segmentation, which provides a full-view and dense-pixel understanding of surroundings in a holistic way. Panoramic segmentation is under-explored due to two critical challenges: (1) image distortions and object deformations on panoramas; (2) lack of annotations for training panoramic segmenters. To tackle these problems, we propose a Transformer for Panoramic Semantic Segmentation (Trans4PASS) architecture. First, to enhance distortion awareness, Trans4PASS, equipped with Deformable Patch Embedding (DPE) and Deformable MLP (DMLP) modules, is capable of handling object deformations and image distortions whenever (before or after adaptation) and wherever (shallow or deep levels) by design. We further introduce the upgraded Trans4PASS+ model, featuring DMLPv2 with parallel token mixing to improve the flexibility and generalizability in modeling discriminative cues. Second, we propose a Mutual Prototypical Adaptation (MPA) strategy for unsupervised domain adaptation. Third, aside from Pinhole-to-Panoramic (PIN2PAN) adaptation, we create a new dataset (SynPASS) with 9,080 panoramic images to explore a Synthetic-to-Real (SYN2REAL) adaptation scheme in 360 • imagery. Extensive experiments are conducted, which cover indoor and outdoor scenarios, and each of them is investigated with PIN2PAN and SYN2REAL regimens. Trans4PASS+ achieves state-of-the-art performances on four domain adaptive panoramic semantic segmentation benchmarks. Code is available at https://github.com/jamycheung/Trans4PASS.

show abstract

DensePASS: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation with Attention-Augmented Context Exchange

Cited by 27 publications

References 18 publications

Review on Panoramic Imaging and Its Applications in Scene Understanding

Review on Panoramic Imaging and Its Applications in Scene Understanding

Is my Driver Observation Model Overconfident? Input-guided Calibration Networks for Reliable and Interpretable Confidence Estimates

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

Contact Info

Product

Resources

About