HMLFC: Hierarchical Motion‐Compensated Light Field Compression for Interactive Rendering

Abstract

We present a new motion‐compensated hierarchical compression scheme (HMLFC) for encoding light field images (LFI) that is suitable for interactive rendering. Our method combines two different approaches, motion compensation schemes and hierarchical compression methods, to exploit redundancies in LFI. The motion compensation schemes capture the redundancies in local regions of the LFI efficiently (local coherence) and the hierarchical schemes capture the redundancies present across the entire LFI (global coherence). Our hybrid approach combines the two schemes effectively capturing both local as well as global coherence to improve the overall compression rate. We compute a tree from LFI using a hierarchical scheme and use phase shifted motion compensation techniques at each level of the hierarchy. Our representation provides random access to the pixel values of the light field, which makes it suitable for interactive rendering applications using a small run‐time memory footprint. Our approach is GPU friendly and allows parallel decoding of LF pixel values. We highlight the performance on the two‐plane parameterized light fields and obtain a compression ratio of 30–800× with a PSNR of 40–45 dB. Overall, we observe a ∼2–5× improvement in compression rates using HMLFC over prior light field compression schemes that provide random access capability. In practice, our algorithm can render new views of resolution 512 × 512 on an NVIDIA GTX‐980 at ∼200 fps.

Neural Subspaces for Light Fields

“…Prior research has augmented this analytical approach with disparity [5], [6], [7] and geometry information [8]. Some sophisticated applications of light field video [9], [10] also integrate motion prediction [11] and build on existing video codec algorithms such as HEVC (H.265) [12] and VP9 [13]. More recently, Le Pendu et al [14] propose a Fourier Disparity Layer representation specific for sub-aperture light fields, which allows upsampling [15] and compression [16], [17] in the Fourier domain.…”

“…Specifically, they use the standard JPEG2000 to encode the representative key view and Bounded Integer Sequence Encoding [26] for the sparse residual views. Pratapa et al [11] later extend this hierarchical method to incorporate the motion prediction and compensation concept widely used in traditional video encoding techniques.…”

“…Compressing high-resolution scenes with such a structural layout is not straightforward for prior methods. Many prior methods for light field compression are tailored to light fields with viewpoints restricted to a 2D planar grid [11], [25], [41] . The non-planar layout of the cameras results in both translation and rotation among the viewpoints, which is ill-posed for these compression methods that utilize translation between viewpoints to represent repetitive patterns compactly.…”

Advanced Scalability for Light Field Image Coding

“…RAp T otal amount of encode bits to decode the f ull light f ield (5) In PVS-based light field coding solutions, RRA p is equal to 1, which means to access a view, the whole encoded light field should be transmitted and the whole bitstream should be decoded (to access, e.g., the last view). In encoding light fields, some compression methods focus on improving random access to arbitrary views [39], [40], [41], [42], [36], [43], [37].…”

RLFC

“…Compared to motion compensation schemes our approach is complimentary and involves a different set of steps to exploit redundancies in LFI to achieve good compression ratios. It would be useful to explore techniques to combine our hierarchical method with the motion compensation techniques to further improve the compression ratios [Pratapa and Manocha 2018]. However, including motion compensation techniques into our scheme may add more overhead to the decoding algorithm and we need to evaluate the pros and cons carefully.…”