Light Field Neural Rendering

Suhail, Mohammed; Esteves, Carlos; Sigal, Leonid; Makadia, Ameesh

doi:10.1109/cvpr52688.2022.00809

Cited by 93 publications

(34 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent works on light field technology cover the topics of angular enhancement [ 100 , 101 , 102 ] (which aims to achieve super-resolution), image enhancement [ 103 , 104 , 105 ] (which is also often named super-resolution, and thus, the terminology of the other one is becoming “angular super-resolution”), saliency detection [ 106 , 107 , 108 ], light field rendering [ 109 , 110 , 111 ] and reconstruction [ 112 , 113 , 114 ], microscopy [ 115 , 116 , 117 ], camera animation [ 118 ], video streaming [ 119 ], objective quality assessment [ 120 , 121 ], perceived quality [ 94 , 122 ], and many more. Every single experiment that studies the projection-based light field visualization with the involvement of human individuals is summarized in a recent work [ 123 ], along with a thorough analysis of future research efforts, and an up-to-date survey [ 124 ] investigates the associated methods.…”

Section: Projection-based Light Field Visualization Technologymentioning

confidence: 99%

How I Met Your V2X Sensor Data: Analysis of Projection-Based Light Field Visualization for Vehicle-to-Everything Communication Protocols and Use Cases

Kara

Wippelhauser

Balogh

et al. 2023

Sensors

View full text Add to dashboard Cite

The practical usage of V2X communication protocols started emerging in recent years. Data built on sensor information are displayed via onboard units and smart devices. However, perceptually obtaining such data may be counterproductive in terms of visual attention, particularly in the case of safety-related applications. Using the windshield as a display may solve this issue, but switching between 2D information and the 3D reality of traffic may introduce issues of its own. To overcome such difficulties, automotive light field visualization is introduced. In this paper, we investigate the visualization of V2X communication protocols and use cases via projection-based light field technology. Our work is motivated by the abundance of V2X sensor data, the low latency of V2X data transfer, the availability of automotive light field prototypes, the prevalent dominance of non-autonomous and non-remote driving, and the lack of V2X-based light field solutions. As our primary contributions, we provide a comprehensive technological review of light field and V2X communication, a set of recommendations for design and implementation, an extensive discussion and implication analysis, the exploration of utilization based on standardized protocols, and use-case-specific considerations.

show abstract

Section: Projection-based Light Field Visualization Technologymentioning

confidence: 99%

How I Met Your V2X Sensor Data: Analysis of Projection-Based Light Field Visualization for Vehicle-to-Everything Communication Protocols and Use Cases

Kara

Wippelhauser

Balogh

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

“…Classical image-based rendering techniques use approximate scene geometry to reproject and blend source image content onto novel views [10,37,46]. Recent works leverage the power of deep learning and neural fields [62] to improve image-based rendering from both structured (e.g., light fields [16,21]) and unstructured data [7,50]. Rather than performing image-based rendering, which requires storing the input images, another approach is to optimize some 3D scene representation augmented with appearance information [41].…”

Section: Related Workmentioning

confidence: 99%

HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling

Attal¹,

Huang²,

Richardt³

et al. 2023

Preprint

View full text Add to dashboard Cite

real-world scenes. To address these issues, we present Hy-perReel -a novel 6-DoF video representation. The two core components of HyperReel are: (1) a ray-conditioned sample prediction network that enables high-fidelity, high frame rate rendering at high resolutions and (2) a compact and memoryefficient dynamic volume representation. Our 6-DoF video pipeline achieves the best performance compared to prior and contemporary approaches in terms of visual quality with small memory requirements, while also rendering at up to 18 frames-per-second at megapixel resolution without any custom CUDA code.

show abstract

“…In this method, pseudo EPIs from unstructured LFs were used in the training process. Suhail et al [37] proposed a two-stage transformer-based model. In this method, the features were first aggregated along the epipolar line dimension and then aggregated along the reference view dimension to produce color information.…”

Section: B Depth-independent Lf Reconstructionmentioning

confidence: 99%

High-Fidelity Light Field Reconstruction Method Using View-Selective Angular Feature Extraction

et al. 2023

View full text Add to dashboard Cite

Deep learning (DL) provides an effective approach for light field (LF) reconstruction that aims to synthesize novel views from sparse-sampled views. However, it is challenging to address domain asymmetry when adopting spatial-angular interaction LF reconstruction methods. To solve this problem, a view-selective angular feature extraction block (VS-LFAFE) is proposed to obtain full-resolution angular features that enumerate whole view directions in a macropixel. By applying the VS-LFAFE, a novel LF reconstruction method is proposed with two subblocks: a spatial-angular feature extraction and fusion block and an angular upsampling block. Experimental results validate that the VS-LFAFE is effective, and the proposed method can achieve superior performance compared with the state-of-the-art methods.

show abstract

Light Field Neural Rendering

Cited by 93 publications

References 48 publications

How I Met Your V2X Sensor Data: Analysis of Projection-Based Light Field Visualization for Vehicle-to-Everything Communication Protocols and Use Cases

How I Met Your V2X Sensor Data: Analysis of Projection-Based Light Field Visualization for Vehicle-to-Everything Communication Protocols and Use Cases

HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling

High-Fidelity Light Field Reconstruction Method Using View-Selective Angular Feature Extraction

Contact Info

Product

Resources

About