Light field reconstruction using hierarchical features fusion

Hu, Zexi; Chung, Yuk Ying; Ouyang, Wanli; Chen, Xiaoming; Chen, Zhibo

doi:10.1016/j.eswa.2020.113394

Cited by 13 publications

(5 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Meng et al [39] proposed a highdimensional convolution-based method, which consisted of a residual network that restored local spatio-angular information and a refinement network that reconstructed the spatial details of the scenes. Hu et al proposed [40] a CNN-based method in which U-Net [41] was used to extract the hierarchical features, and spatio-angular separable (SAS) convolution layers were used to separate and fuse the spatial and angular features. By applying the spatial-angular alternating mechanism, this method can be trained on larger patches and can improve performance particularly in occluded regions.…”

Section: B Depth-independent Lf Reconstructionmentioning

confidence: 99%

High-Fidelity Light Field Reconstruction Method Using View-Selective Angular Feature Extraction

et al. 2023

View full text Add to dashboard Cite

Deep learning (DL) provides an effective approach for light field (LF) reconstruction that aims to synthesize novel views from sparse-sampled views. However, it is challenging to address domain asymmetry when adopting spatial-angular interaction LF reconstruction methods. To solve this problem, a view-selective angular feature extraction block (VS-LFAFE) is proposed to obtain full-resolution angular features that enumerate whole view directions in a macropixel. By applying the VS-LFAFE, a novel LF reconstruction method is proposed with two subblocks: a spatial-angular feature extraction and fusion block and an angular upsampling block. Experimental results validate that the VS-LFAFE is effective, and the proposed method can achieve superior performance compared with the state-of-the-art methods.

show abstract

Section: B Depth-independent Lf Reconstructionmentioning

confidence: 99%

High-Fidelity Light Field Reconstruction Method Using View-Selective Angular Feature Extraction

et al. 2023

View full text Add to dashboard Cite

show abstract

“…They wasted angular data by only using horizontal or vertical EPI stacks. Hu et al [ 14 ] proposed LF reconstruction with hierarchical feature fusion. SAS layers were employed to extract features from 4D LF images, while the U-Net structure was adopted to generate both semantic and local feature representation.…”

Section: Related Workmentioning

confidence: 99%

“…The LF reconstruction challenge reconstructs dense LF images from sparse input views. Previous approaches using the convolutional neural network (CNN) without depth estimation [ 13 , 14 ] can only handle LFs with a small baseline. They explore the connection between the angular and spatial domains but fail to use the epipolar information fully.…”

Section: Introductionmentioning

confidence: 99%

RCA-LF: Dense Light Field Reconstruction Using Residual Channel Attention Networks

Salem

Ibrahem

Kang

2022

Sensors

View full text Add to dashboard Cite

Dense multi-view image reconstruction has played an active role in research for a long time and interest has recently increased. Multi-view images can solve many problems and enhance the efficiency of many applications. This paper presents a more specific solution for reconstructing high-density light field (LF) images. We present this solution for images captured by Lytro Illum cameras to solve the implicit problem related to the discrepancy between angular and spatial resolution resulting from poor sensor resolution. We introduce the residual channel attention light field (RCA-LF) structure to solve different LF reconstruction tasks. In our approach, view images are grouped in one stack where epipolar information is available. We use 2D convolution layers to process and extract features from the stacked view images. Our method adopts the channel attention mechanism to learn the relation between different views and give higher weight to the most important features, restoring more texture details. Finally, experimental results indicate that the proposed model outperforms earlier state-of-the-art methods for visual and numerical evaluation.

show abstract

“…LF fusion [10] and Depthguided techniques [11] have been popular in reconstructing an LF from a single or a sparse set of SAIs. Hu et allet@tokeneonedot [12] aimed for a faster LF reconstruction method by using hierarchical features fusion.…”

Section: Light Field View Synthesismentioning

confidence: 99%

Light Field Compression by Residual CNN Assisted JPEG

Hedayati,

Havens,

Bos

2020

Preprint

View full text Add to dashboard Cite

Light field (LF) imaging has gained significant attention due to its recent success in 3-dimensional (3D) displaying and rendering as well as augmented and virtual reality usage. Nonetheless, because of the two extra dimensions, LFs are much larger than conventional images. We develop a JPEG-assisted learning-based technique to reconstruct an LF from a JPEG bitstream with a bit per pixel ratio of 0.0047 on average. For compression, we keep the LF's center view and use JPEG compression with 50% quality. Our reconstruction pipeline consists of a small JPEG enhancement network (JPEG-Hance), a depth estimation network (Depth-Net), followed by view synthesizing by warping the enhanced center view. Our pipeline is significantly faster than using video compression on pseudosequences extracted from an LF, both in compression and decompression, while maintaining effective performance. We show that with a 1% compression time cost and 18x speedup for decompression, our methods reconstructed LFs have better structural similarity index metric (SSIM) and comparable peak signal-to-noise ratio (PSNR) compared to the state-of-the-art video compression techniques used to compress LFs.1 Introduction(to be completed based on the length) Light fields (LF) have two extra dimensions, as compared to conventional images, which represents angular information of the scene. Hence, LFs contain a large volume of data that makes storing and portability time consuming and costly. Also, decompressing LF video with a high angular resolution at acceptable frames per second(fps) for streaming is challenging. We aim to address these problems by predicting the entire LF from its JPEG compressed center view.Direct application of the standard image compression techniques, such as JPEG, PNG, etclet@tokeneonedot, on the LF do not take advantage of the existing redundancy between LF views. A better success has been achieved from the use of video compression techniques. For using video compression methods on LFs, a sequence of images has been build from LF views which is called pseudo-sequence [1]. A combination of machine learning (ML) methods, capable of predicting LF views, and video compression techniques, has been explored in [2]. In this work, we present a combination of JPEG compression with ML view predictions. LF synthesis techniques have shown the possibility of estimating the entire LF from a single or a set of sparse views. Here, we show that there is enough information in the JPEG compressed center view-as well as a group of sub-aperture images (SAI)-to predict the entire LF with sufficient quality. We test the success of our method by comparing against state-of-the-art methods in LF compression that use the existing HEVC compression.Our method is faster in compression and decompression by 100x and 10x, respectively, compared to the direct use of HEVC. This speed up means a set of 30 LFs with a spatial resolution of (375, 540) and angular resolution of (7, 7) can be decompressed on a typical gaming GPU in less than 0.02 second, while HEVC-base...

show abstract

Light field reconstruction using hierarchical features fusion

Cited by 13 publications

References 10 publications

High-Fidelity Light Field Reconstruction Method Using View-Selective Angular Feature Extraction

High-Fidelity Light Field Reconstruction Method Using View-Selective Angular Feature Extraction

RCA-LF: Dense Light Field Reconstruction Using Residual Channel Attention Networks

Light Field Compression by Residual CNN Assisted JPEG

Contact Info

Product

Resources

About