Interactive Pixel‐Accurate Free Viewpoint Rendering from Images with Silhouette Aware Sampling

Abstract

We present an integrated, fully GPU-based processing pipeline to interactively render new views of arbitrary scenes from calibrated but otherwise unstructured input views. In a two-step procedure, our method first generates for each input view a dense proxy of the scene using a new multi-view stereo formulation. Each scene proxy consists of a structured cloud of feature aware particles which automatically have their image space footprints aligned to depth discontinuities of the scene geometry and hence effectively handle sharp object boundaries and occlusions. We propose a particle optimization routine combined with a special parameterization of the view space which enables an efficient proxy generation as well as robust and intuitive filter operators for noise and outlier removal. Moreover, our generic proxy generation allows us to flexibly handle scene complexities ranging from small objects up to complete outdoor scenes. The second phase of the algorithm combines these particle clouds in real-time into a view-dependent proxy for the desired output view and performs a pixel-accurate accumulation of the color contributions from each available input view. This makes it possible to reconstruct even fine-scale view-dependent illumination effects. We demonstrate how all these processing stages of the pipeline can be implemented entirely on the GPU with memory efficient, scalable data structures for maximum performance. This allows us to generate new output renderings of high visual quality from input images in real-time.

Real-time high-resolution sparse voxelization with application to image-based modeling

“…Using a particle-based passive stereo approach, [Hornung and Kobbelt 2009] developed an interactive system for free viewpoint rendering of a static scene from a collection of pre-processed images. Kinect Fusion builds a voxel model of a static scene in realtime using a single Kinect depth sensor that is moved through space [Izadi et al 2011].…”

3D shape and texture morphing using 2D projection and reconstruction

“…The point cloud will contain potentially noisy data, as well as duplicate points for regions that are visible in multiple images. The point cloud is filtered and smoothed in a manner similar to [27] to eliminate these issues. Finally Poisson surface reconstruction is applied to the point cloud to form the geometry, G α for the morph stage.…”

Video-based characters

“…One way of synthesizing textures in novel views or poses would be to use some form of projective texturing and blending, e.g., [Debevec et al 1996;Narayanan et al 1998;Buehler et al 2001;Carranza et al 2003;Cheung 2003;Hornung and Kobbelt 2009]. However, these approaches are known to produce texture ghosting when there are even the slightest of inaccuracies in scene geometry.…”