Multiview video: Acquisition, processing, compression, and virtual view rendering

Stankiewicz, Olgierd; Lafruit, Gauthier; Domański, Marek

doi:10.1016/b978-0-12-811889-4.00001-4

Cited by 21 publications

(16 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The depth information represents the distance between the camera and the objects in the scene, in "scene units" defined in the specification. In order to better suit the human visual system, depth information is typically stored as normalized disparity instead of distance in meters [7]. Fig.…”

Section: M I V C O D E C O V E R V I E W a Source Materialsmentioning

confidence: 99%

MPEG Immersive Video Coding Standard

et al. 2021

View full text Add to dashboard Cite

This article introduces the ISO/IEC MPEG Immersive Video (MIV) standard, MPEG-I Part 12, which is undergoing standardization. The draft MIV standard provides support for viewing immersive volumetric content captured by multiple cameras with six degrees of freedom (6DoF) within a viewing space that is determined by the camera arrangement in the capture rig. The bitstream format and decoding processes of the draft specification along with aspects of the Test Model for Immersive Video (TMIV) reference software encoder, decoder, and renderer are described. The use cases, test conditions, quality assessment methods, and experimental results are provided. In the TMIV, multiple texture and geometry views are coded as atlases of patches using a legacy 2-D video codec, while optimizing for bitrate, pixel rate, and quality. The design of the bitstream format and decoder is based on the visual volumetric video-based coding (V3C) and video-based point cloud compression (V-PCC) standard, MPEG-I Part 5.

show abstract

Section: M I V C O D E C O V E R V I E W a Source Materialsmentioning

confidence: 99%

MPEG Immersive Video Coding Standard

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Therefore, increasing the coding efficiency of depth maps of 6DoF videos is crucial for providing an immersive visual experience. Some studies proposed the reduction of the depth information errors that may occur in the representation of depth with a finite number of bits intended for better coding efficiency [17], [18].…”

Section: Introductionmentioning

confidence: 99%

Nonlinear Depth Quantization Using Piecewise Linear Scaling for Immersive Video Coding

Park

Lim

et al. 2022

IEEE Access

View full text Add to dashboard Cite

Moving Picture Experts Group (MPEG) is developing a standard for immersive video coding called MPEG Immersive Video (MIV) and is releasing a reference software called Test Model for Immersive Video (TMIV) in the standardization process. The TMIV efficiently compresses an immersive video comprising a set of texture and depth views acquired using multiple cameras within a limited 3D viewing space. Moreover, it affords a rendered view of an arbitrary view position and orientation with six degrees of freedom (6DoF). However, the existing depth quantization applied to depth atlas in TMIV is insufficient since the reconstructed depth is crucial for achieving the required quality of a rendered viewport. To address this issue, we propose a nonlinear depth quantization method that allocates more codewords to a depth subrange with a higher occurrence of depth values located at edge regions, which are important in terms of the rendered view quality. We implement the proposed nonlinear quantization based on piecewise linear scaling considering the computational complexity and bitstream overhead. The experimental results show that the proposed method yields PSNR-based Bjøntegaard delta rate gains of 5.2% and 4.9% in the end-toend performance for High-and Low-bitrate (BR) ranges, respectively. Moreover, subjective quality improvement is mainly observed at the object boundaries of the rendered viewport. The proposed nonlinear quantization method has been adopted into the TMIV as a candidate standard technology for the next MIV edition.

show abstract

“…Thermal noise [11] • • environment, electronics Salt and pepper noise [7] • • electronics Random telegraph noise [4] • • electronics Temporal contrast/ brightness inconsistencies [12] • • electronics, environment, software homomorphic filtering [13], stabilization algorithms [14], temporal filtering [12], neural networks [15] Line, stripe, wave and ring artifacts [16,17] • • electronics, environment, optics wavelet/Fourier filtering [10], spatial filtering [16], neural networks [18] Compression artifacts [19] • • software bilateral filtering [8], fuzzy filtering [20] neural networks [19,[21][22][23] Projective distortions [24] • • optics model-based calculations [25], neural networks [26,27] Out-of-focus effects [28,29] • • optics morphological filtering [30], neural networks [31,32] Fixed pattern noise [33,34] • • electronics, environment, optics reference imaging [33], neural networks [35] Aliasing [36] • • software anti-aliasing algorithms [36], neural networks [37] Rolling shutter effects [38] • • electronics neural networks [39] Artifacts are visually recognizable in a variety of shapes and intensities. Table 1 shows common artifact types occurring in sensor images, their sources, and algorithmic example methods which can be used to...…”

Section: Introductionmentioning

confidence: 99%

A Data-Centric Augmentation Approach for Disturbed Sensor Image Segmentation

2021

View full text Add to dashboard Cite

Deep learning methods have become increasingly popular for optical sensor image analysis. They are adaptable to specific tasks and simultaneously demonstrate a high degree of generalization capability. However, applying deep neural networks to problems with low availability of labeled training data can lead to a model being incapable of generalizing to possible scenarios that may occur in test data, especially with the occurrence of dominant imaging artifacts. We propose a data-centric augmentation approach based on generative adversarial networks that overlays the existing labeled data with synthetic artifacts that are generated from data not present in the training set. This augmentation leads to a more robust generalization capability in semantic segmentation. Our method does not need any additional labeling and does not lead to additional memory or time consumption during inference. Further, we find it to be more effective than comparable approaches that are based on procedurally generated disturbances and the direct use of real disturbances. Building upon the improved segmentation results, we observe that our approach leads to improvements of 22% in the F1-score for an evaluated detection problem, which promises significant robustness towards future disturbances. In the context of sensor-based data analysis, the compensation of image artifacts is a challenge. When the structures of interest are not clearly visible in an image, algorithms that can cope with artifacts are crucial for obtaining the desired information. Thereby, the high variation of artifacts, the combination of different types of artifacts, and their similarity to signals of interest are specific issues that have to be considered in the analysis. Despite the high generalization capability of deep learning-based approaches, their recent success was driven by the availability of large amounts of labeled data. Therefore, the provision of comprehensive labeled image data with different characteristics of image artifacts is of importance. At the same time, applying deep neural networks to problems with low availability of labeled data remains a challenge. This work presents a data-centric augmentation approach based on generative adversarial networks that augments the existing labeled data with synthetic artifacts generated from data not present in the training set. In our experiments, this augmentation leads to a more robust generalization in segmentation. Our method does not need additional labeling and does not lead to additional memory or time consumption during inference. Further, we find it to be more effective than comparable augmentations based on procedurally generated artifacts and the direct use of real artifacts. Building upon the improved segmentation results, we observe that our approach leads to improvements of 22% in the F1-score for an evaluated detection problem. Having achieved these results with an example sensor, we expect increased robustness against artifacts in future applications.

show abstract

Multiview video: Acquisition, processing, compression, and virtual view rendering

Cited by 21 publications

References 46 publications

MPEG Immersive Video Coding Standard

MPEG Immersive Video Coding Standard

Nonlinear Depth Quantization Using Piecewise Linear Scaling for Immersive Video Coding

A Data-Centric Augmentation Approach for Disturbed Sensor Image Segmentation

Contact Info

Product

Resources

About