Distortion Convolution Module for Semantic Segmentation of Panoramic Images Based on the Image-Forming Principle

Hu, Xiaozhi; An, Yi; Shao, Cheng; Hu, Huosheng

doi:10.1109/tim.2021.3139710

Cited by 16 publications

(5 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The distortion convolutional module (DCM) is introduced in the network encoder by Hu et al [36]. The module is developed to correct the image distortion according to the image-forming principle.…”

Section: Panoramic Semantic Segmentationmentioning

confidence: 99%

Real-Time Semantic Segmentation of Spherical Images for Automotive Applications

Dell'Eva

Marco

Cerri

et al. 2023

Preprint

View full text Add to dashboard Cite

Recent advancements in autonomous driving technology have resulted in a growing need for robust algorithms that can effectively detect, recognize, and segment objects in the surrounding environment. Semantic segmentation systems, which classify each pixel of an image into a class, have become a key component in achieving complete scene understanding, a crucial task for self-driving vehicles. This work focuses on utilizing deep learning methods to perform semantic segmentation on spherical images, a particular representation of fisheye images, characterized by a large field of view and strong distortion, which can result in performance degradation on convolutional neural networks (CNNs) trained on pinhole images. To address this issue, we propose the use of spherical kernels, which adapt their sampling locations according to the spherical projection, as an alternative to standard convolutional kernels, in real-time CNNs. We demonstrate that this approach outperforms the use of the regular convolution operation. To showcase the effectiveness of this method, we generate three spherical image datasets with varying characteristics.

show abstract

Section: Panoramic Semantic Segmentationmentioning

confidence: 99%

Real-Time Semantic Segmentation of Spherical Images for Automotive Applications

Dell'Eva

Marco

Cerri

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Distortions in panoramic images have been taken into considerations. Hu et al [7] considered the panoramic image generation process and tackled distortions via a deformableconvolution-based module. Zhang et al [264] developed a distortion-aware transformer model for adapting to panoramic images.…”

Section: A Semantic Scene Understanding With Image Segmentationmentioning

confidence: 99%

“…However, the above requirements usually entail addressing different trade-offs, which make the design of panoramic imaging instruments particularly challenging. Compared with the only previous panoramic imaging survey [1], it is so far back that it does not provide an overview of the flourishing progress in the field of panoramic imaging in the last 20 years [2]- [7], which is addressed in this review. Surveys on other sensing have emerged, such as LiDAR [8], [9] or applications in scene understanding [10]- [12].…”

Section: Introductionmentioning

confidence: 99%

Review on Panoramic Imaging and Its Applications in Scene Understanding

Gao¹,

Yang²,

Sheng³

et al. 2022

Preprint

View full text Add to dashboard Cite

With the rapid development of high-speed communication and artificial intelligence technologies, human perception of real-world scenes is no longer limited to the use of small Field of View (FoV) and low-dimensional scene detection devices. Panoramic imaging emerges as the next generation of innovative intelligent instruments for environmental perception and measurement. However, while satisfying the need for large-FoV photographic imaging, panoramic imaging instruments are expected to have high resolution, no blind area, miniaturization, and multi-dimensional intelligent perception, and can be combined with artificial intelligence methods towards the next generation of intelligent instruments, enabling deeper understanding and more holistic perception of 360 • real-world surrounding environments. Fortunately, recent advances in freeform surfaces, thinplate optics, and metasurfaces provide innovative approaches to address human perception of the environment, offering promising ideas beyond conventional optical imaging. In this review, we begin with introducing the basic principles of panoramic imaging systems, and then describe the architectures, features, and functions of various panoramic imaging systems. Afterwards, we discuss in detail the broad application prospects and great design potential of freeform surfaces, thin-plate optics, and metasurfaces in panoramic imaging. We then provide a detailed analysis on how these techniques can help enhance the performance of panoramic imaging systems. We further offer a detailed analysis of applications of panoramic imaging in scene understanding for autonomous driving and robotics, spanning panoramic semantic image segmentation, panoramic depth estimation, panoramic visual localization, and so on. Finally, we cast a perspective on future potential and research directions for panoramic imaging instruments.

show abstract

“…Capturing wide-FoV scenes, panoramic images [4] act as a starting point for a more complete scene understanding. Mainstream outdoor omnidirectional semantic segmentation systems rely on fisheye cameras [57], [58], [59] or panoramic images [60], [61], [62]. Panoramic panoptic segmentation is also addressed in recent surrounding parsing systems [63], [64], [65], where the video segmentation pipeline with the Waymo open dataset [64] has a coverage of 220 • .…”

Section: Panoramic Segmentationmentioning

confidence: 99%

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

Zhang¹,

Yang²,

Sheng³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we address panoramic semantic segmentation, which provides a full-view and dense-pixel understanding of surroundings in a holistic way. Panoramic segmentation is under-explored due to two critical challenges: (1) image distortions and object deformations on panoramas; (2) lack of annotations for training panoramic segmenters. To tackle these problems, we propose a Transformer for Panoramic Semantic Segmentation (Trans4PASS) architecture. First, to enhance distortion awareness, Trans4PASS, equipped with Deformable Patch Embedding (DPE) and Deformable MLP (DMLP) modules, is capable of handling object deformations and image distortions whenever (before or after adaptation) and wherever (shallow or deep levels) by design. We further introduce the upgraded Trans4PASS+ model, featuring DMLPv2 with parallel token mixing to improve the flexibility and generalizability in modeling discriminative cues. Second, we propose a Mutual Prototypical Adaptation (MPA) strategy for unsupervised domain adaptation. Third, aside from Pinhole-to-Panoramic (PIN2PAN) adaptation, we create a new dataset (SynPASS) with 9,080 panoramic images to explore a Synthetic-to-Real (SYN2REAL) adaptation scheme in 360 • imagery. Extensive experiments are conducted, which cover indoor and outdoor scenarios, and each of them is investigated with PIN2PAN and SYN2REAL regimens. Trans4PASS+ achieves state-of-the-art performances on four domain adaptive panoramic semantic segmentation benchmarks. Code is available at https://github.com/jamycheung/Trans4PASS.

show abstract

Distortion Convolution Module for Semantic Segmentation of Panoramic Images Based on the Image-Forming Principle

Cited by 16 publications

References 45 publications

Real-Time Semantic Segmentation of Spherical Images for Automotive Applications

Real-Time Semantic Segmentation of Spherical Images for Automotive Applications

Review on Panoramic Imaging and Its Applications in Scene Understanding

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

Contact Info

Product

Resources

About