Review on Panoramic Imaging and Its Applications in Scene Understanding

Gao, Shaohua; Yang, Kailun; Sheng, Hao; Wang, Kaiwei; Bai, Jing

doi:10.48550/arxiv.2205.05570

Cited by 7 publications

(10 citation statements)

References 266 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Capturing wide-FoV scenes, panoramic images [4] act as a starting point for a more complete scene understanding. Mainstream outdoor omnidirectional semantic segmentation systems rely on fisheye cameras [57], [58], [59] or panoramic images [60], [61], [62].…”

Section: Panoramic Segmentationmentioning

confidence: 99%

“…P ANORAMIC semantic segmentation offers an omnidirectional and dense visual understanding regimen that integrates 360 • perception of surrounding scenes and pixel-wise predictions of input images [1]. The attracted attention of 360 • cameras are manifesting, with an increasing number of learning systems and practical applications, such as holistic sensing in autonomous vehicles [2], [3], [4] and immersive viewing in augmented-and virtual reality (AR/VR) devices [5], [6], [7]. In contrast to images captured via pinhole cameras (Fig.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

Zhang¹,

Yang²,

Sheng³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we address panoramic semantic segmentation, which provides a full-view and dense-pixel understanding of surroundings in a holistic way. Panoramic segmentation is under-explored due to two critical challenges: (1) image distortions and object deformations on panoramas; (2) lack of annotations for training panoramic segmenters. To tackle these problems, we propose a Transformer for Panoramic Semantic Segmentation (Trans4PASS) architecture. First, to enhance distortion awareness, Trans4PASS, equipped with Deformable Patch Embedding (DPE) and Deformable MLP (DMLP) modules, is capable of handling object deformations and image distortions whenever (before or after adaptation) and wherever (shallow or deep levels) by design. We further introduce the upgraded Trans4PASS+ model, featuring DMLPv2 with parallel token mixing to improve the flexibility and generalizability in modeling discriminative cues. Second, we propose a Mutual Prototypical Adaptation (MPA) strategy for unsupervised domain adaptation. Third, aside from Pinhole-to-Panoramic (PIN2PAN) adaptation, we create a new dataset (SynPASS) with 9,080 panoramic images to explore a Synthetic-to-Real (SYN2REAL) adaptation scheme in 360 • imagery. Extensive experiments are conducted, which cover indoor and outdoor scenarios, and each of them is investigated with PIN2PAN and SYN2REAL regimens. Trans4PASS+ achieves state-of-the-art performances on four domain adaptive panoramic semantic segmentation benchmarks. Code is available at https://github.com/jamycheung/Trans4PASS.

show abstract

Section: Panoramic Segmentationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

Zhang¹,

Yang²,

Sheng³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…To this intent, we introduce Panoramic Panoptic Segmentation, to enable complete image understanding based on standard camera input. Our task combines panoptic image level understanding and the wide FoV of panoramic images [12], tackling both previously discussed problems in a cost minimizing way.…”

Section: Robust Model Baseline Modelmentioning

confidence: 99%

Panoramic Panoptic Segmentation: Towards Complete Surrounding Understanding via Unsupervised Contrastive Learning

Jaus

Yang

Stiefelhagen

2021

2021 IEEE Intelligent Vehicles Symposium (IV)

Self Cite

View full text Add to dashboard Cite

In this work, we introduce panoramic panoptic segmentation as the most holistic scene understanding both in terms of field of view and image level understanding. A complete surrounding understanding provides a maximum of information to the agent, which is essential for any intelligent vehicle in order to make informed decisions in a safetycritical dynamic environment such as real-world traffic. In order to overcome the lack of annotated panoramic images, we propose a framework which allows model training on standard pinhole images and transfers the learned features to a different domain. Using our proposed method, we manage to achieve significant improvements of over 5% measured in PQ over nonadapted models on our Wild Panoramic Panoptic Segmentation (WildPPS) dataset. We show that our proposed Panoramic Robust Feature (PRF) framework is not only suitable to improve performance on panoramic images but can be beneficial whenever model training and deployment are executed on data taken from different distributions. As an additional contribution, we publish WildPPS: The first panoramic panoptic image dataset to foster progress in surrounding perception.

show abstract

“…I MAGES with very large Field of View (FoV) are attracting more attention on account of more abundant information they provide about the surrounding environment for some Computer Vision (CV) tasks [1]. A considerable amount of research is carried out on semantic segmentation [2]- [4], object detection [5], [6], visual SLAM [7]- [9] and other CV tasks [10]- [12] under images with 360 • FoV, namely panoramic CV tasks.…”

Section: Introductionmentioning

confidence: 99%

Annular Computational Imaging: Capture Clear Panoramic Images through Simple Lens

Jiang¹,

Sheng²,

Sun³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Panoramic Annular Lens (PAL), composed of few lenses, has great potential in panoramic surrounding sensing tasks for mobile and wearable devices because of its tiny size and large Field of View (FoV). However, the image quality of tiny-volume PAL confines to optical limit due to the lack of lenses for aberration correction. In this paper, we propose an Annular Computational Imaging (ACI) framework to break the optical limit of light-weight PAL design. To facilitate learning-based image restoration, we introduce a wave-based simulation pipeline for panoramic imaging and tackle the synthetic-to-real gap through multiple data distributions. The proposed pipeline can be easily adapted to any PAL with design parameters and is suitable for loose-tolerance designs. Furthermore, we design the Physics Informed Image Restoration Network (PI 2 RNet), considering the physical priors of panoramic imaging and physics-informed learning. At the dataset level, we create the DIVPano dataset and the extensive experiments on it illustrate that our proposed network sets the new state of the art in the panoramic image restoration under spatially-variant degradation. In addition, the evaluation of the proposed ACI on a simple PAL with only 3 spherical lenses reveals the delicate balance between high-quality panoramic imaging and compact design. To the best of our knowledge, we are the first to explore Computational Imaging (CI) in PAL. Code and datasets will be made publicly available at https://github.com/zju-jiangqi/ACI-PI2RNet.

show abstract

Review on Panoramic Imaging and Its Applications in Scene Understanding

Cited by 7 publications

References 266 publications

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

Panoramic Panoptic Segmentation: Towards Complete Surrounding Understanding via Unsupervised Contrastive Learning

Annular Computational Imaging: Capture Clear Panoramic Images through Simple Lens

Contact Info

Product

Resources

About