Self-Supervised Light Field Reconstruction Using Shearlet Transform and Cycle Consistency

Gao, Yuan; Bregović, Robert; Gotchev, Atanas

doi:10.1109/lsp.2020.3008082

Cited by 4 publications

(5 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Guo et al [75] with three residual blocks to restore high-frequency information on EPIs. The work in [76] first transformed the EPI to the shearlet domain and then used an encoder-decoder generative adversarial network (GAN) to estimate the residual information and reconstruct the shearlet coefficients. Finally, the coefficient is transformed back to the image domain.…”

Section: Epi Super-resolutionmentioning

confidence: 99%

Learning-based light field imaging: an overview

Mahmoudpour,

Pagliari,

Schelkens

2024

J Image Video Proc.

View full text Add to dashboard Cite

Conventional photography can only provide a two-dimensional image of the scene, whereas emerging imaging modalities such as light field enable the representation of higher dimensional visual information by capturing light rays from different directions. Light fields provide immersive experiences, a sense of presence in the scene, and can enhance different vision tasks. Hence, research into light field processing methods has become increasingly popular. It does, however, come at the cost of higher data volume and computational complexity. With the growing deployment of machine-learning and deep architectures in image processing applications, a paradigm shift toward learning-based approaches has also been observed in the design of light field processing methods. Various learning-based approaches are developed to process the high volume of light field data efficiently for different vision tasks while improving performance. Taking into account the diversity of light field vision tasks and the deployed learning-based frameworks, it is necessary to survey the scattered learning-based works in the domain to gain insight into the current trends and challenges. This paper aims to review the existing learning-based solutions for light field imaging and to summarize the most promising frameworks. Moreover, evaluation methods and available light field datasets are highlighted. Lastly, the review concludes with a brief outlook for future research directions.

show abstract

Section: Epi Super-resolutionmentioning

confidence: 99%

Learning-based light field imaging: an overview

Mahmoudpour,

Pagliari,

Schelkens

2024

J Image Video Proc.

View full text Add to dashboard Cite

show abstract

“…Vagharshakyan et al in [26] proposed using an adapted discrete shearlet transform to sparse the light field and derived a method of light field reconstruction in the directionally sensitive transform domain. Similar methods and results are also introduced in reference [27]. In [28], Shi et al applied the continuous frequency domain of a light field to sparse light field signals and reconstruct novel views that typically work in the discrete Fourier domain.…”

Section: A Reconstruction Using Spectral Analysismentioning

confidence: 99%

“…(27) By (27), we can calculate the phase spectrum of the light field. We use a sine function to construct a light field with…”

Section: Proof Of Theoremmentioning

confidence: 99%

Multiview Reconstruction Using Phase Retrieval of Light Field Signals

He¹,

Zhang²,

Zhu

2022

IEEE Access

View full text Add to dashboard Cite

Light field rendering is an effective multi-view reconstruction technology. The current multiview reconstruction methods still has the problem of low view synthesis quality in complex scenes, such as occlusion and non Lambertian surfaces. In this article, we consider undistorted multiview reconstruction and develop a phase retrieval method for light field (PRLF) signals to improve the rendering quality. Our approach utilizes the phase spectrum in both the time domain and frequency domain of the discrete light field because the phase of the light field carries more scene information. The phase corrections are "light field truncation" in the time domain and "phase replacement" in the frequency domain. The iterative algorithm used provides high-quality estimated results for the relative phase of the light field. By the estimated phase results, we derive a multiview reconstruction method to improve the rendering quality. Finally, to verify the claimed performance, we also compare the PRLF with the most advanced light field reconstruction algorithms. The experimental results show that the proposed PRLF outperforms other known reconstruction schemes.INDEX TERMS Light field rendering, phase retrieval, novel view reconstruction, rendering quality I. INTRODUCTION

show abstract

“…In terms of the problem of DSLF reconstruction, it is extremely difficult to capture ground-truth DSLF training data as introduced in the beginning of this thesis (Section 1.1). To resolve this problem, Gao et al propose a novel self-supervised DSLF reconstruction approach, referred to as CycleST, of which the network can be trained solely on synthetic SSLF data [GBG20].…”

Section: Light Field Novel View Synthesismentioning

confidence: 99%

“…To produce the final reconstructed target DSLF, we need to compensate for the shearing parameter ϕ, applied in the previous pre-shearing step, for all the reconstructed densely-sampled EPIs. More details can also be found in [GBG20].…”

Section: Shearlet Transform (St)mentioning

confidence: 99%

Large-Scale Light Field Capture and Reconstruction

Yuan¹

Self Cite

View full text Add to dashboard Cite

This thesis discusses approaches and techniques to convert Sparsely- Sampled Light Fields (SSLFs) into Densely-Sampled Light Fields (DSLFs), which can be used for visualization on 3DTV and Virtual Reality (VR) de- vices. Exemplarily, a movable 1D large-scale light ﬁeld acquisition system for capturing SSLFs in real-world environments is evaluated. This system consists of 24 sparsely placed RGB cameras and two Kinect V2 sensors. The real-world SSLF data captured with this setup can be leveraged to reconstruct real-world DSLFs. To this end, three challenging problems require to be solved for this system: (i) how to estimate the rigid trans- formation from the coordinate system of a Kinect V2 to the coordinate system of an RGB camera; (ii) how to register the two Kinect V2 sensors with a large displacement; (iii) how to reconstruct a DSLF from a SSLF with moderate and large disparity ranges. To overcome these three challenges, we propose: (i) a novel self- calibration method, which takes advantage of the geometric constraints from the scene and the cameras, for estimating the rigid transformations from the camera coordinate frame of one Kinect V2 to the camera coordi- nate frames of 12-nearest RGB cameras; (ii) a novel coarse-to-ﬁne approach for recovering the rigid transformation from the coordinate system of one Kinect to the coordinate system of the other by means of local color and geometry information; (iii) several novel algorithms that can be categorized into two groups for reconstructing a DSLF from an input SSLF, including novel view synthesis methods, which are inspired by the state-of-the-art video frame interpolation algorithms, and Epipolar-Plane Image (EPI) in- painting methods, which are inspired by the Shearlet Transform (ST)-based DSLF reconstruction approaches.

show abstract

Self-Supervised Light Field Reconstruction Using Shearlet Transform and Cycle Consistency

Cited by 4 publications

References 43 publications

Learning-based light field imaging: an overview

Learning-based light field imaging: an overview

Multiview Reconstruction Using Phase Retrieval of Light Field Signals

Large-Scale Light Field Capture and Reconstruction

Contact Info

Product

Resources

About