Steered mixture-of-experts for light field coding, depth estimation, and processing

Abstract

The proposed framework, called Steered Mixture-ofExperts (SMoE), enables a multitude of processing tasks on light fields using a single unified Bayesian model. The underlying assumption is that light field rays are instantiations of a non-linear or non-stationary random process that can be modeled by piecewise stationary processes in the spatial domain. As such, it is modeled as a space-continuous Gaussian Mixture Model. Consequently, the model takes into account different regions of the scene, their edges, and their development along the spatial and disparity dimensions.Applications presented include light field coding, depth estimation, edge detection, segmentation, and view interpolation. The representation is compact, which allows for very efficient compression yielding state-of-the-art coding results for low bit-rates. Furthermore, due to the statistical representation, a vast amount of information can be queried from the model even without having to analyze the pixel values. This allows for "blind" light field processing and classification Index Terms-light field coding, depth estimation, light field representations, mixture-of-experts, mixture models

Deep-learning-based macro-pixel synthesis and lossless coding of light field images

“…In the lossy compression domain, LF coding is an important topic which triggered a lot of interest in the compression community and in standardization bodies such as JPEG [30] and MPEG [31]. Several solutions were proposed by modifying the HEVC standard [13] to take into account the specific nature of the plenoptic image [14][15][16][17][18]. In [19], the authors propose a method for scalable lossy-to-lossless coding based on depth information.…”

Steered mixture-of-experts for light field video coding

“…The coordinate space is 2D, 3D, 4D, and 5D in the case of respectively images, video, static and dynamic light fields. [4][5][6] The color space for color images is conventionally represented as a 3D space, e.g. RGB or YCbCr.…”

“…The parameters are decorrelated, quantized and further binarized by using an arithmetic coder. 6,11 For the remainder of this paper, we will provide the revised coding methodology applied on SMoE models of light field video, without loss of generality. Finally, we will compare our proposed scheme with MV-HEVC.…”

“…(a) Sort kernels based on a greedy fashion along centers µ j in order to minimize the distance between two consecutive kernel centers (b) Transform parameters to be laplacian distributed per parameter (c) Merge all distributions on the same laplacian distribution with variances according to the importance of the parameter (d) Quantize the values uniformly (e) Encode using a laplacian adaptive arithmetic coder 3.1 Coordinate covariance R X j X j quantization As in related work, 6,11 we employ a vector quantization-like method for coding the window covariance R Xj Xj . We propose a minibatch EM-like algorithm based on the Kullback-Leibler (KL) divergence.…”

“…Currently, SMoE has been successfully applied for images, video, and static 4D light fields for coding, with competitive rate-distortion (RD) results for low-to mid-level bitrates. [4][5][6] The local regressors currently reported are only linear and thus modeling very high spatial frequencies is challenging, however, the theory does not limit the nature of the local regressors and further developments are an active area of research. Recently, work has been published on the modeling and rendering of SMoE models for light field video, however, without coding.…”