Toward Real-Time and Efficient Compression of Human Time-Varying Meshes

Abstract

In this paper, a novel skeleton-based approach to human time-varying mesh (H-TVM) compression is presented. The topic of TVM compression is new and has many challenges, such as handling the lack of obvious mapping of vertices across frames and handling the variable connectivity across frames, while maintaining efficiency, which are the most important ones. Very few works exist in the literature, while not all of the challenges have been addressed yet. In addition, developing an efficient and real-time solution, handling the above, obviously is a difficult task. We attempt to address the H-TVM compression problem inspired from video coding using different types of frames and trying to efficiently remove inter-frame geometric redundancy utilizing the recent advances in human skeleton tracking. The overall approach focuses on compression efficiency, low distortion, and low computation time enabling for realtime transmission of H-TVMs. It efficiently compresses geometry and vertex attributes of TVMs. In addition, this paper is the first to provide an efficient method for connectivity coding of TVMs, by introducing a modification to the state-of-the-art MPEG-4 TFAN algorithm. Experiments are conducted in the MPEG-3DGC TVM database. The method outperforms the stateof-the-art standardized static mesh coder MPEG-4 TFAN at low bit-rates, while remaining competent at high bit-rates. It gives a practical proof of concept that in the combined problem of geometry, connectivity, and vertex attribute coding of TVMs, efficient inter-frame redundancy removal is possible, establishing ground for further improvements. Finally, this paper proposes a method for motion-based coding of H-TVMs that can further enhance the overall experience when H-TVM compression is used in a tele-immersion scenario.

A Design and Evaluation Framework for a Tele-Immersive Mixed Reality Platform

“…In the real world, outdoor users interact via augmented reality presentations on a mobile device and wearable sensors. Two deployments exist for indoor users using virtual reality, in order to assess the impact of different configurations on UX: a 'high end' configuration which includes a CAVE environment, wearable motion sensors, dedicated simulators and rendering of full-body reconstruction of remote users [17]. The consumer level or 'low end' configuration displays the virtual environment on a large, conventional display and integrates Nintendo's Wii for simulation, while simple avatars represent the remote users.…”

A pipeline for multiparty volumetric video conferencing

“…Wu et al [12] present a psychophysical approach to Color-plus-Depth Level of Detail for polygon mesh based 3D tele immersive video. Doumanoglou et al [3] propose a skeleton based approach to compress human time varying meshes by modifying the MPEG4 TFAN codec. In this work we use real time photo realistic point cloud reconstructions of users along with low latency DASH.…”

An Integrated Platform for Live 3D Human Reconstruction and Motion Capturing

“…To enable fast transmission in the real-time mode and efficient storage in the quick-post one, an intra-frame compression scheme (JPEG for the RGB and LZ4 entropy compression for the depth images) was employed due to its reduced complexity and processing time. These modes enable either: 1) online 3D reconstruction, thus making it suitable for TI applications when combined with real-time efficient data compression [2] or 2) temporally complete and higher quality results, exploiting all the recorded data in a quick-post processing step.…”