Light Field Image Coding Based on Hybrid Data Representation

Abstract

This paper proposes a novel efficient light field coding approach based on a hybrid data representation. Current state-of-the-art light field coding solutions either operate on micro-images or sub-aperture images. Consequently, the intrinsic redundancy that exists in light field images is not fully exploited, as is demonstrated. This novel hybrid data representation approach allows to simultaneously exploit four types of redundancies: i) sub-aperture image intra spatial redundancy, ii) sub-aperture image inter-view redundancy, iii) intra-micro-image redundancy, and iv) inter-micro-image redundancy between neighboring micro-images. The proposed light field coding solution allows flexibility for several types of baselines, by adaptively exploiting the most predominant type of redundancy on a coding block basis. To demonstrate the efficiency of using a hybrid representation, this paper proposes a set of efficient pixel prediction methods combined with a pseudo-video sequence coding approach, based on the HEVC standard. Experimental results show consistent average bitrate savings when the proposed codec is compared to relevant state-ofthe-art benchmarks. For lenslet light field content, the proposed coding algorithm outperforms the HEVCbased pseudo-video sequence coding benchmark by an average bitrate savings of 23%. It is shown for the same light field content that the proposed solution outperforms JPEG Pleno verification models MuLE and WaSP, as these codecs are only able to achieve 11% and −14% bitrate savings over the same HEVC-based benchmark, respectively. The performance of the proposed coding approach is also validated for light fields with wider baselines, captured with high-density camera arrays, being able to outperform both the HEVCbased benchmark, as well as MuLE and WaSP.INDEX TERMS Light field representation, light field image coding, HEVC, pseudo-video sequence, spatial pixel prediction, least squares prediction.

Supporting Wider Baseline Light Fields in JPEG Pleno With a Novel Slanted 4D-DCT Coding Mode

“…Different scanning patterns have been proposed to improve the RD performance by better exploiting the redundancy between MIs or PIs; good examples can be found in [22], [23]. In [24], an HEVC-based codec mixes hybrid prediction structures for MIs and PIs, depending on which is more efficient for different parts of the light field. In [25], light field image views are grouped into the so-called Macro View Images (MVIs), which are groups of independent 3 × 3 views.…”

The JPEG Pleno Light Field Coding Standard 4D-Transform Mode: How to Design an Efficient 4D-Native Codec

“…To maximize the RD performance by better exploiting the redundancy between MIs or PIs, multiple video coding prediction structures and scanning patterns have been proposed in the literature for rearranging the LF data into a pseudo-video. More recently, a HEVC-based LF codec, labelled HEVC-HR, has been proposed which mixes hybrid prediction structures for MIs or PIs, depending on which is more efficient for different parts of the LF [23].…”

“…• HEVC anchor (HEVC), which corresponds to the pseudo-video like HEVC coding of the 13×13 views in a IPPP serpentine scanning pattern (left-right, topdown) using the x265 [54] implementation [11]; this is the solution taken as reference for all BD-Rate values in this paper. • 4D-Prediction mode (WaSP), which corresponds to the other LF coding mode specified in the JPEG Pleno Light Field Coding standard [10], using the JPEG Pleno VM2.1 Verification Model software [51]; • HEVC-HR, which corresponds to the LF codec specified in [23] and reviewed in Section II; • MV-HEVC-based, which corresponds to the LF codec specified in [28] and reviewed in Section II; • VVC-serpentine, which corresponds to the pseudovideo-like VVC coding of the 13×13 views in a IPPP serpentine (left-right, top-down) scanning pattern [11]; while more optimized scannings could be used, serpentine scanning is used here to follow the JPEG Pleno CTC recommendations as for the HEVC anchor. Due to constraints of the VVC reference software [55], and to allow a fairer comparison, the LF views were cropped to 624× 432.…”

“…The authors would like to thank the authors of [23], [38], [39] for sharing their results and the authors of [19] for providing their configuration files.…”