A Study on the Influence of Omnidirectional Distortion on CNN-based Stereo Vision

Seuffert, Julian; Grassi, Ana Pérez; Scheck, Tobias; Hirtz, Gangolf

doi:10.5220/0010324808090816

Cited by 6 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, wide support for all required projections is not common, as most of the work in depth estimation is focusing on perspective binocular stereo rigs [20]. Depth estimation from ERP views is also commonly researched [29], [30], [31], but both projections are rarely supported simultaneously, especially for deep learning methods where learning from both perspective and omnidirectional images does not provide satisfactory quality [72]. Nevertheless, since real-world multimedia systems are usually designed for a particular application, wide support for different types of input views is not so crucial outside the standardization process.…”

Section: B Depth Estimation For Immersive Videomentioning

confidence: 99%

Overview and Efficiency of Decoder-Side Depth Estimation in MPEG Immersive Video

Mieloch¹,

Garus

Milovanović

et al. 2022

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

This paper presents the overview and rationale behind the Decoder-Side Depth Estimation (DSDE) mode of the MPEG Immersive Video (MIV) standard, using the Geometry Absent profile, for efficient compression of immersive multiview video. A MIV bitstream generated by an encoder operating in the DSDE mode does not include depth maps. It only contains the information required to reconstruct them in the client or in the cloud: decoded views and metadata. The paper explains the technical details and techniques supported by this novel MIV DSDE mode. The description additionally includes the specification on Geometry Assistance Supplemental Enhancement Information which helps to reduce the complexity of depth estimation, when performed in the cloud or at the decoder side. The depth estimation in MIV is a non-normative part of the decoding process, therefore, any method can be used to compute the depth maps. This paper lists a set of requirements for depth estimation, induced by the specific characteristics of the DSDE. The depth estimation reference software, continuously and collaboratively developed with MIV to meet these requirements, is presented in this paper. Several original experimental results are presented. The efficiency of the DSDE is compared to two MIV profiles. The combined non-transmission of depth maps and efficient coding of textures enabled by the DSDE leads to efficient compression and rendering quality improvement compared to the usual encoder-side depth estimation. Moreover, results of the first evaluation of state-of-the-art multiview depth estimators in the DSDE context, including machine learning techniques, are presented.

show abstract

Section: B Depth Estimation For Immersive Videomentioning

confidence: 99%

Overview and Efficiency of Decoder-Side Depth Estimation in MPEG Immersive Video

Mieloch¹,

Garus

Milovanović

et al. 2022

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

show abstract

“…Therefore, stereo matching on the ODIs is more similar to the human vision system. In [156], they discussed the influence of omnidirectional distortion on the CNN-based methods and compared the quality of disparity maps predicted from the perspective and omnidirectional stereo images. The experimental results show that stereo matching based on the ODIs is more advantageous for numerous applications, e.g., robotics, AR/VR, and several other applications.…”

Section: Stereo Matchingmentioning

confidence: 99%

Deep Learning for Omnidirectional Vision: A Survey and New Perspectives

Ai¹,

Cao²,

Zhu³

et al. 2022

Preprint

View full text Add to dashboard Cite

To whom it may concern,In this paper, we provide an in-depth and comprehensive survey on recent technical advancements in deep learning for omnidirectional vision, and also provide new perspectives for the future direction. Deep learning (DL) has recently been applied to omnidirectional vision. DL-based omnidirectional vision methods often achieve the state-of-the-art (SoTA) performance on various benchmark datasets. Diverse deep neural network (DNN) models have been developed, ranging from the convolutions neural network (CNN)-based models to vision transformers (ViT)-based ones. Generally, the SoTA DNN-based methods differ from each other in four major aspects: convolutional filters used to extract features from the omnidirectional image (ODI) data, network design considering the input numbers and projection types, omnidirectional vision with novel learning strategies, and practical applications. In this paper, we provide a comprehensive and systematic review and analysis of the recent progress in DL methods for omnidirectional vision. There are some previous surveys in the literature. However, some of them are focused on the specific vision tasks, especially room layout reconstruction, 3D scene geometry recovery. Moreover, some provide limited reviews of omnidirectional video streaming methods. Unlike existing surveys, this paper highlights the importance of deep learning and probe the recent advances for omnidirectional vision, both methodically and comprehensively. To the best of our knowledge, this is the first survey to comprehensively review and analyze the DL methods for omnidirectional vision. The main contributions of this paper to the community are five folds: (I) We comprehensively review and analyze the DL methods for omnidirectional vision, including the omnidirectional imaging principle, representation learning, datasets, a taxonomy, and applications, to highlight the differences and difficulties with the 2D planner image data. (II) We conduct an analytical study of recent trends of DL for omnidirectional vision, both hierarchically and structurally. Moreover, we offer insights into the discussion and challenge of each category. (III) We summarize the latest novel learning strategies and potential applications for omnidirectional vision.(I) We provide insightful discussions of the challenges and open problems yet to be solved and propose the potential future directions to spur more in-depth research by the community.(II) We create an open-source repository that provides a taxonomy of all the mentioned works and code links, and hope it can shed light on future research. The organization of this paper is structured as follows. In Sec.2, we introduce the imaging principle of ODI, convolution methods for omnidirectional vision, and some representative datasets. In Sec.3, we review and analyze the existing DL approaches for various tasks and provide taxonomies to categorize the relevant papers. Sec.4 covers novel learning paradigms for the tasks in omnidirectional vision, e.g., unsupervised learning, transfer learning, and reinforcement learning. Sec.5 then scrutinizes the applications, followed by Sec. 6, where we discuss open problems and future directions. Finally, we conclude this paper in Sec. 7. Furthermore, we summarize most, if not all but representative, works (over 200 papers) in the last five years, which were published in the top-tier conferences and journals in computer vision/graphics and machine learning. Due to the lack of space, we show the experimental results in Sec. 2 of the suppl. material. We believe that this paper will bring significant interest in such crucial topics in the community and provide fundamental technical references for further research.

show abstract

OmniGlasses: an optical aid for stereo vision CNNs to enable omnidirectional image processing

Seuffert,

Perez Grassi,

Ahmed

et al. 2024

Machine Vision and Applications

View full text Add to dashboard Cite

Stereo vision is a key technology for 3D scene reconstruction from image pairs. Most approaches process perspective images from commodity cameras. These images, however, have a very limited field of view and only picture a small portion of the scene. In contrast, omnidirectional images, also known as fisheye images, exhibit a much larger field of view and allow a full 3D scene reconstruction with a small amount of cameras if placed carefully. However, omnidirectional images are strongly distorted which make the 3D reconstruction much more sophisticated. Nowadays, a lot of research is conducted on CNNs for omnidirectional stereo vision. Nevertheless, a significant gap between estimation accuracy and throughput can be observed in the literature. This work aims to bridge this gap by introducing a novel set of transformations, namely OmniGlasses. These are incorporated into the architecture of a fast network, i.e., AnyNet, originally designed for scene reconstruction on perspective images. Our network, Omni-AnyNet, produces accurate omnidirectional distance maps with a mean absolute error of around 13 cm at 48.4 fps and is therefore real-time capable.

show abstract

A Study on the Influence of Omnidirectional Distortion on CNN-based Stereo Vision

Cited by 6 publications

References 0 publications

Overview and Efficiency of Decoder-Side Depth Estimation in MPEG Immersive Video

Overview and Efficiency of Decoder-Side Depth Estimation in MPEG Immersive Video

Deep Learning for Omnidirectional Vision: A Survey and New Perspectives

OmniGlasses: an optical aid for stereo vision CNNs to enable omnidirectional image processing

Contact Info

Product

Resources

About