A 3D tele-immersion system based on live captured mesh geometry

Abstract

3D Tele-immersion enables participants in remote locations to share, in real-time, an activity. It offers users natural interactivity and immersive experiences, but it challenges current networking solutions. Work in the past has mainly focused on the efficient delivery of image-based 3D videos and on the realistic rendering and reconstruction of geometry-based 3D objects. The contribution of this paper is a complete media pipeline that allows for geometry-based 3D tele-immersion. Unlike previous approaches, that stream videos or video plus depth estimate, our streaming module can transmit the live-reconstructed 3D representations (triangle meshes). Based on a set of comparative experiments, this paper details the architecture and describes a novel component that can efficiently stream geometry in realtime. This component includes both a novel fast local compression algorithm and a rateless packet protection scheme geared towards the requirements imposed by real-time transmission of live-capture mesh geometry. Tests on a large dataset show an encoding and decoding speed-up of over 10 times at similar compression and quality rates, when compared to the high-end MPEG-4 SC3DMC mesh encoder. The implemented rateless code ensures complete packet loss protection of the triangle mesh object and avoids delay introduced by retransmissions. This approach is compared to a streaming mechanism over TCP and outperforms it at packet loss rates over 2% and/or latencies over 9 ms in terms of end-to-end transmission delay. As reported in this paper, the component has been successfully integrated into a larger tele-immersive environment that includes beyond state of the art 3D reconstruction and rendering modules. This resulted in a prototype that can capture, compress transmit and render triangle mesh geometry in real-time over the internet.

Dynamic adaptive mesh streaming for real-time 3D teleimmersion

“…In that work, depth maps are directly captured using multiple Microsoft Kinects but the compression and transmission scheme is not studied. Another TI system presented in [Mekuria et al 2013] uses the 3D reconstruction scheme of [Alexiadis et al 2013a] but the mesh connectivity coding is restricted to a specific triangulation rule. Additionally, in all the aforementioned works the mesh color is coded per vertex.…”

REVERIE: Natural human interaction in virtual immersive environments

“…3D content coding, transmission and synchronization technologies are being developed and integrated to support the specific 3D data types effectively through the network [4,5,6,7]. The framework distinguishes between semi-static (avatars, off-line user generated objects, videos) and real-time content (3D audio, reconstructed 3D visual/motion data and scene/clock sync messages).…”

A framework for realistic 3D tele-immersion

“…It achieves a compact representation for each reconstructed mesh. Tests on a large dataset show an encoding and decoding speed-up of over 10 to 20 times at similar compression and quality rates, when compared to the high-end MPEG-4 SC3DMC mesh encoder [21]. Generally, mesh geometry transmission over lossy networks has been challenging, as triangle mesh compression have not been designed with resilience to losses in mind.…”

“…This implemented rateless code, similar to fountain codes [19], ensures complete packet loss protection of the triangle mesh object and avoids delay introduced by retransmissions. This approach is compared to a streaming mechanism over TCP and outperforms it at packet loss rates over 2% and/or latencies over 9ms in terms of end-to-end transmission delay [21].…”