Hard Real-Time, Pixel-Parallel Rendering of Light Field Videos Using Steered Mixture-of-Experts

Abstract

Steered Mixture-of-Experts (SMoE) is a novel framework for the approximation, coding, and description of image modalities such as light field images and video. The future goal is to arrive at a representation for Six Degrees-of-Freedom (6DoF) image data. Previous research has shown the feasibility of real-time pixel-parallel rendering of static light field images. Each pixel is independently reconstructed by kernels that lay in its vicinity. The number of kernels involved forms the bottleneck on the achievable framerate. The goal of this paper is twofold. Firstly, we introduce pixel-level rendering of light field video, as previous work only rendered static content. Secondly, we investigate rendering using a predefined number of most significant kernels. As such, we can deliver hard real-time constraints by trading off the reconstruction quality.

Steered mixture-of-experts for light field video coding

“…Recently, work has been published on the modeling and rendering of SMoE models for light field video, however, without coding. 7,8 The goal of this paper is to revise and generalize previous SMoE coding methods and compare with the state-of-the-art in terms of RD-performance.…”

“…Evidently, SMoE has proven its potential by presenting beneficial properties for the distribution of 6DoF visual content compared to traditional image coding methods. [7][8][9][10][11] For instance, in the case of rendering it has three important properties. Firstly, view-rendering is very lightweight and pixels are coded independent from one another.…”