SLAMCast: Large-Scale, Real-Time 3D Reconstruction and Streaming for Immersive Multi-Client Live Telepresence

Abstract

Fig. 1. Illustration of our novel multi-client live telepresence framework for remote collaboration: RGB-D data captured with consumergrade cameras represent the input to our real-time large-scale reconstruction technique that is based on a novel thread-safe GPU hash map data structure. Efficient data streaming is achieved by transmitting a novel compact representation of the reconstructed model in terms of Marching Cubes indices. Multi-client live telepresence is achieved by the server's independent handling of client requests.Abstract-Real-time 3D scene reconstruction from RGB-D sensor data, as well as the exploration of such data in VR/AR settings, has seen tremendous progress in recent years. The combination of both these components into telepresence systems, however, comes with significant technical challenges. All approaches proposed so far are extremely demanding on input and output devices, compute resources and transmission bandwidth, and they do not reach the level of immediacy required for applications such as remote collaboration. Here, we introduce what we believe is the first practical client-server system for real-time capture and many-user exploration of static 3D scenes. Our system is based on the observation that interactive frame rates are sufficient for capturing and reconstruction, and real-time performance is only required on the client site to achieve lag-free view updates when rendering the 3D model. Starting from this insight, we extend previous voxel block hashing frameworks by introducing a novel thread-safe GPU hash map data structure that is robust under massively concurrent retrieval, insertion and removal of entries on a thread level. We further propose a novel transmission scheme for volume data that is specifically targeted to Marching Cubes geometry reconstruction and enables a 90% reduction in bandwidth between server and exploration clients. The resulting system poses very moderate requirements on network bandwidth, latency and client-side computation, which enables it to rely entirely on consumer-grade hardware, including mobile devices. We demonstrate that our technique achieves state-of-the-art representation accuracy while providing, for any number of clients, an immersive and fluid lag-free viewing experience even during network outages.

Efficient 3D Reconstruction and Streaming for Group-Scale Multi-Client Live Telepresence

“…constraints regarding latency and bandwidth. We demonstrate that our extensions result in compact high-quality 3D reconstructions and finally allow the immersion of more than 24 people within the same live-captured scene (beyond room-scale), thereby significantly exceeding the number of immersed persons in previous approaches [27] without adding or exchanging hardware components. In summary, the key contributions of this work are (1) an efficient novel set of filters designed to optimize the performance and scalability of current state-of-the-art telepresence systems at the example of the SLAMCast system [27], (2) an adaption of the telepresencespecific filters to standalone volumetric 3D reconstruction and (3) a comprehensive evaluation of the beneficial effect of the proposed set of filters regarding scalability, latency and visual quality.…”

“…In contrast, interactive telepresence for individual remote users within live-captured static scenes has been addressed by Mossel and Kröter [20] based on voxel block hash-ing [12,23]. The limitations of this system regarding high bandwidth requirements, the immersion of only a single remote user into the captured scenarios as well as network interruptions leading to loss of scene parts that are reconstructed in the meantime have been overcome in the recent SLAMCast system [27]. However, the scalability to immersing large groups of people into on-the-fly captured scenes has not been achieved so far.…”

“…Previous teleconferencing systems [4,24,28] were designed to capture a fixed region of interest based on expensive well-calibrated acquisition setups involving statically mounted cameras. In contrast, the live telepresence system by Stotko et al [27] is tailored to the acquisition of scenes beyond such a fixed size defined by the setup and involves portable, consumer-grade capture hardware. As a result, efficiently representing and transmitting the scene to visualization devices is significantly harder.…”

A VR System for Immersive Teleoperation and Live Exploration with a Mobile Robot

“…the set of updated voxel blocks that need to be streamed to the individual client. For efficient streaming to the clients, the received TSDF voxel blocks are converted to the bandwidthefficient MC voxel block representation [2] and then added to the stream sets of each connected exploration client. Here, we used a simplified version of the Marching Cubes (MC) technique [41] where the weights have been discarded.…”

“…Due to the significant progress in VR displays in recent years, the immersive exploration of scenes based on virtual reality systems has gained a lot of attention with diverse applications in entertainment, teleconferencing [1], remote collaboration [2], medical rehabilitation and education. The quality of immersive experience of places, while being physically located in another environment, opens new opportunities for robotic teleoperation scenarios.…”