Highly parallel steered mixture-of-experts rendering at pixel-level for image and light field data

Abstract

A novel image approximation framework called Steered Mixture-of-Experts (SMoE) was recently presented. SMoE has multiple applications in coding, scale-conversion, and general processing of image modalities. In particular, it has strong potential for coding and streaming higher dimensional image modalities that are necessary to leverage full translational and rotational freedom (6 Degrees-of-Freedom) in virtual reality for camera captured images. In this paper, we analyze the rendering performance of SMoE for 2D images and 4D light fields. Two different GPU implementations that parallelize the SMoE regression step at pixel-level are presented, including experimental evaluations based on rendering performance and quality. In this paper it is shown that on appropriate hardware, the OpenCL implementation can achieve 85fps and 22fps for respectively 1080p and 4K renderings of large models with more than 100.000 of Gaussian kernels.

Hard Real-Time, Pixel-Parallel Rendering of Light Field Videos Using Steered Mixture-of-Experts

“…Given a set of Gaussian parameters, the goal of this architecture is to reconstruct 2D views which are planar slices of the 5D coordinate space. This architecture consists of two levels of parallelization, similar to [10]. The first level is on block-level, the second level is performed on pixel-level.…”

“…Especially for light field images, SMoE was shown to yield competitive rate-distortion results for low-to mid-level bitrates [8]. For rendering it has three important properties [10]. Firstly, view-rendering is lightweight and pixel-parallel.…”

“…Previous research has shown that real-time rendering of light field images is possible, given appropriate hardware [10]. However, the framerate is heavily dependent on the number of kernels that are involved for the calculation of a single pixel.…”

Progressive Modeling of Steered Mixture-of-Experts for Light Field Video Approximation

“…It thus presents the decoder with MPEG-7 like descriptors including spatial edges, intensity flow, motion (video), and depth (light fields) [6]- [9]. For rendering it has three important properties [10]. Firstly, view-rendering is lightweight and pixel-parallel.…”

Steered mixture-of-experts for light field video coding

“…Evidently, SMoE has proven its potential by presenting beneficial properties for the distribution of 6DoF visual content compared to traditional image coding methods. [7][8][9][10][11] For instance, in the case of rendering it has three important properties. Firstly, view-rendering is very lightweight and pixels are coded independent from one another.…”