Transfer Beyond the Field of View: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation

Zhang, Jiaming; Ma, Chaoxiang; Yang, Kailun; Roitberg, Alina; Peng, Kunyu; Stiefelhagen, Rainer

doi:10.1109/tits.2021.3123070

Cited by 33 publications

(10 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our proposed method outperforms recent state-of-the-art token mixing [22], [23], [24], [25], deformable patchbased learning [26], transformer domain adaptation [27], and panoramic segmentation [3], [9], [10], [21] methods. ⊕ On four panoramic datasets, our framework yields superior results, spanning indoor and outdoor scenarios, before and after PIN2PAN and SYN2REAL domain adaptation.…”

Section: Introductionmentioning

confidence: 88%

“…On the indoor synthetic Structured3D dataset [20], our SYN2REAL-adapted model surpasses the model trained using extra 1,400 target data. On the outdoor DensePASS dataset [3], our Trans4PASS+ model obtains 50.23% in mIoU with a +11.21% gain over the baseline source-only model without adaptation, and our Pin2Pan-adapted model obtains 57.23% in mIoU with a +15.24% boost over the previous best method [21].…”

Section: Introductionmentioning

confidence: 90%

“…To confirm the generalization capacity of applying Trans4PASS in adaptation methods, FANet [121] and DANet [12] used in P2PDA [3] (a) Per-class results on DensePASS. Comparison with state-of-the-art panoramic segmentation [9], [10], domain adaptation [16], [17], [21], [27], [101], [106], and multi-supervision methods [129], [130], [131]. * denotes performing Multi-Scale (MS) evaluation.…”

Section: Ablation Of Unsupervised Domain Adaptationmentioning

confidence: 99%

See 2 more Smart Citations

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

Zhang¹,

Yang²,

Sheng³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we address panoramic semantic segmentation, which provides a full-view and dense-pixel understanding of surroundings in a holistic way. Panoramic segmentation is under-explored due to two critical challenges: (1) image distortions and object deformations on panoramas; (2) lack of annotations for training panoramic segmenters. To tackle these problems, we propose a Transformer for Panoramic Semantic Segmentation (Trans4PASS) architecture. First, to enhance distortion awareness, Trans4PASS, equipped with Deformable Patch Embedding (DPE) and Deformable MLP (DMLP) modules, is capable of handling object deformations and image distortions whenever (before or after adaptation) and wherever (shallow or deep levels) by design. We further introduce the upgraded Trans4PASS+ model, featuring DMLPv2 with parallel token mixing to improve the flexibility and generalizability in modeling discriminative cues. Second, we propose a Mutual Prototypical Adaptation (MPA) strategy for unsupervised domain adaptation. Third, aside from Pinhole-to-Panoramic (PIN2PAN) adaptation, we create a new dataset (SynPASS) with 9,080 panoramic images to explore a Synthetic-to-Real (SYN2REAL) adaptation scheme in 360 • imagery. Extensive experiments are conducted, which cover indoor and outdoor scenarios, and each of them is investigated with PIN2PAN and SYN2REAL regimens. Trans4PASS+ achieves state-of-the-art performances on four domain adaptive panoramic semantic segmentation benchmarks. Code is available at https://github.com/jamycheung/Trans4PASS.

show abstract

Section: Introductionmentioning

confidence: 88%

Section: Introductionmentioning

confidence: 90%

Section: Ablation Of Unsupervised Domain Adaptationmentioning

confidence: 99%

See 1 more Smart Citation

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

Zhang¹,

Yang²,

Sheng³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…For domain generalization, one can only use the source domain images, with the aim to produce a robust, generalized segmentation model in the target domain. P2PDA [253], [254] (Fig. 11(c)) designed attentionaugmented domain adaptation modules to detect and magnify the pinhole-panoramic correspondences in multiple spaces.…”

Section: A Semantic Scene Understanding With Image Segmentationmentioning

confidence: 99%

Review on Panoramic Imaging and Its Applications in Scene Understanding

Gao¹,

Yang²,

Sheng³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

With the rapid development of high-speed communication and artificial intelligence technologies, human perception of real-world scenes is no longer limited to the use of small Field of View (FoV) and low-dimensional scene detection devices. Panoramic imaging emerges as the next generation of innovative intelligent instruments for environmental perception and measurement. However, while satisfying the need for large-FoV photographic imaging, panoramic imaging instruments are expected to have high resolution, no blind area, miniaturization, and multi-dimensional intelligent perception, and can be combined with artificial intelligence methods towards the next generation of intelligent instruments, enabling deeper understanding and more holistic perception of 360 • real-world surrounding environments. Fortunately, recent advances in freeform surfaces, thinplate optics, and metasurfaces provide innovative approaches to address human perception of the environment, offering promising ideas beyond conventional optical imaging. In this review, we begin with introducing the basic principles of panoramic imaging systems, and then describe the architectures, features, and functions of various panoramic imaging systems. Afterwards, we discuss in detail the broad application prospects and great design potential of freeform surfaces, thin-plate optics, and metasurfaces in panoramic imaging. We then provide a detailed analysis on how these techniques can help enhance the performance of panoramic imaging systems. We further offer a detailed analysis of applications of panoramic imaging in scene understanding for autonomous driving and robotics, spanning panoramic semantic image segmentation, panoramic depth estimation, panoramic visual localization, and so on. Finally, we cast a perspective on future potential and research directions for panoramic imaging instruments.

show abstract

“…In [25], contextaware omni-supervised models are taken to the wild, fulfilling panoramic semantic segmentation in a single pass with enhanced generalizability. P2PDA [67] explicitly tackles panoramic segmentation from a domain adaptation perspective by transferring from the label-rich pinhole domain to the labelscarce panoramic domain. In [68], a distortion convolutional module is developed to correct the panoramic image distortion according to the image-forming principle.…”

Section: B Panoramic Scene Segmentationmentioning

confidence: 99%

Panoramic Panoptic Segmentation: Towards Complete Surrounding Understanding via Unsupervised Contrastive Learning

Jaus

Yang

Stiefelhagen

2021

2021 IEEE Intelligent Vehicles Symposium (IV)

Self Cite

View full text Add to dashboard Cite

In this work, we introduce panoramic panoptic segmentation as the most holistic scene understanding both in terms of field of view and image level understanding. A complete surrounding understanding provides a maximum of information to the agent, which is essential for any intelligent vehicle in order to make informed decisions in a safetycritical dynamic environment such as real-world traffic. In order to overcome the lack of annotated panoramic images, we propose a framework which allows model training on standard pinhole images and transfers the learned features to a different domain. Using our proposed method, we manage to achieve significant improvements of over 5% measured in PQ over nonadapted models on our Wild Panoramic Panoptic Segmentation (WildPPS) dataset. We show that our proposed Panoramic Robust Feature (PRF) framework is not only suitable to improve performance on panoramic images but can be beneficial whenever model training and deployment are executed on data taken from different distributions. As an additional contribution, we publish WildPPS: The first panoramic panoptic image dataset to foster progress in surrounding perception.

show abstract

Transfer Beyond the Field of View: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation

Cited by 33 publications

References 67 publications

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

Review on Panoramic Imaging and Its Applications in Scene Understanding

Panoramic Panoptic Segmentation: Towards Complete Surrounding Understanding via Unsupervised Contrastive Learning

Contact Info

Product

Resources

About