A low-cost, practical acquisition and rendering pipeline for real-time free-viewpoint video communication

Abstract

We present a semiautomatic real-time pipeline for capturing and rendering free-viewpoint video using passive stereo matching. The pipeline is simple and achieves agreeable quality in real time on a system of commodity web cameras and a single desktop computer. We suggest an automatic algorithm to compute a constrained search space for an efficient and robust hierarchical stereo reconstruction algorithm. Due to our fast reconstruction times, we can eliminate the need for an expensive global surface reconstruction with a combination of high coverage and aggressive filtering. Finally, we employ a novel color weighting scheme that generates credible new viewpoints without noticeable seams, while keeping the computational complexity low. The simplicity and low cost of the system make it an accessible and more practical alternative for many applications compared to previous methods.

Camera and Lidar-Based View Generation for Augmented Remote Operation in Mining Applications

“…As such, the solutions presented in Sections III through VI are designed for handling the live sensor data within the real-time limits imposed by said sensors. While our system does not achieve as low render times as shown by [20], we avoid some of the limitations imposed therein, such as the greenscreen keying and background removal, and demonstrate an alternate approach that still attains real-time performance within the constraints outlined in [5].…”

“…The 3D model can also be used to create a textured 3D mesh, or to anchor and orient the augmented rendering overlays into the presented view. More recently, free viewpoint rendering is converging with mesh-based rendering for enabling real-time applications with multiple sensors as scene data sources, as seen in [20,21].…”

“…Rasmuson et al [20] present a real-time processing pipeline for free-viewpoint rendering for videoconferencing, with focus on real time capability. They use several nonrectified 2D videocameras and projective geometry to construct a 3D mesh for a central object seen by all cameras.…”

“…Meerits et al [21] triangulate a mesh from recorded dense depth points of RGB-D sensors, instead of estimating a mesh from point correspondences like [20]. The depth from RGB-D sensors is converted to a point cloud and normals are estimated for each point.…”

“…Content may change prior to final publication. camera views still need to be displayed for the operator, and rectification can significantly distort the viewing plane [20], making such views unsuitable for direct presentation.…”