A Flexible Coding Scheme Based on Block Krylov Subspace Approximation for Light Field Displays with Stacked Multiplicative Layers

Abstract

To create a realistic 3D perception on glasses-free displays, it is critical to support continuous motion parallax, greater depths of field, and wider fields of view. A new type of Layered or Tensor light field 3D display has attracted greater attention these days. Using only a few light-attenuating pixelized layers (e.g., LCD panels), it supports many views from different viewing directions that can be displayed simultaneously with a high resolution. This paper presents a novel flexible scheme for efficient layer-based representation and lossy compression of light fields on layered displays. The proposed scheme learns stacked multiplicative layers optimized using a convolutional neural network (CNN). The intrinsic redundancy in light field data is efficiently removed by analyzing the hidden low-rank structure of multiplicative layers on a Krylov subspace. Factorization derived from Block Krylov singular value decomposition (BK-SVD) exploits the spatial correlation in layer patterns for multiplicative layers with varying low ranks. Further, encoding with HEVC eliminates inter-frame and intra-frame redundancies in the low-rank approximated representation of layers and improves the compression efficiency. The scheme is flexible to realize multiple bitrates at the decoder by adjusting the ranks of BK-SVD representation and HEVC quantization. Thus, it would complement the generality and flexibility of a data-driven CNN-based method for coding with multiple bitrates within a single training framework for practical display applications. Extensive experiments demonstrate that the proposed coding scheme achieves substantial bitrate savings compared with pseudo-sequence-based light field compression approaches and state-of-the-art JPEG and HEVC coders.

A Hierarchical Approach for Lossy Light Field Compression With Multiple Bit Rates Based on Tucker Decomposition via Random Sketching

“…Moreover, signal representation in the proposed scheme is more flexible to support extended functionality such as light field view interpolation or extrapolation and is appropriate for more general rendering by shifting approximated SAIs in the depth dimension instead of two angular dimensions. Thus, the proposed formulation differs from our previous work [58].…”

“…In our previous work Ravishankar et al [58], multiplicative layers from complete 13x13 light fields were learnt using a convolutional neural network. Spatial and temporal correlations present among SAIs were taken into account and the hidden low-rank structure of the multiplicative layers was analyzed on a Krylov subspace.…”

Objective Quality Assessment Metrics for Light Field Image Based on Textural Features

“…This metric measures the structural similarity of depth maps between the original and distorted LF to predict the quality of the distorted LF. Regarding the state-of-the-art LF quality assessment methods based on quality scores, there are also several recent studies [30][31][32][33][34] that present the overview of LF-IQA with NR or blind LF approaches based on LF characteristics (i.e., spatial-angular information) and also provide a flexible coding scheme based on block Krylov subspace approximation for LF displays.…”

An integrated learning and approximation scheme for coding of static or dynamic light fields based on hybrid Tucker–Karhunen–Loève transform‐singular value decomposition via tensor double sketching

“…Our previous work on light field coding exploits the spatial and temporal correlations among light field multiplicative layers [40]. We handle inherent redundancies in light fields by approximating multiplicative layers in the image‐based spatial domain.…”