A pipeline for multiparty volumetric video conferencing

Abstract

The advent of affordable 3D capture and display hardware is making volumetric videoconferencing feasible. This technology increases the immersion of the participants, breaking the flat restriction of 2D screens, by allowing them to collaborate and interact in shared virtual reality spaces. In this paper we introduce the design and development of an architecture intended for volumetric videoconferencing that provides a highly realistic 3D representation of the participants, based on pointclouds. A pointcloud representation is suitable for real-time applications like video conferencing, due to its low-complexity and because it does not need a time consuming reconstruction process. As transport protocol we selected low latency DASH, due to its popularity and client-based adaptation mechanisms for tiling. This paper presents the architectural design, details the implementation, and provides some referential results. The demo will showcase the system in action, enabling volumetric videoconferencing using pointclouds. CCS CONCEPTS • Information systems → Multimedia streaming; • Humancentered computing → Interaction paradigms; • Networks → Network protocols.

Mediascape XR

“…The application has been already showcased in the VRDays Europe 2021 3 . Figure 3 shows the specific setup used for that event.…”

“…MediaScape XR is a SocialVR application, build on top of the VRTogether platform [3], which allows the co-presence of multiple users in the same virtual space, a museum in this case. The application is developed in Unity3D.…”

“…On the way to the main artefact of the exhibition, they find a mirror (2) where they can check their photo-realistic representations. Next they find the main artefact, the costume (3). Users are able to read its history and characteristics on the information panels, but also to interact with it, change size and rotate it.…”

Scalable MDC-Based Volumetric Video Delivery for Real-Time One-to-Many WebRTC Conferencing

“…In the scenario of one-on-one interactions, such as video calls, a major requirement is having the lowest possible latency to enable real-time interaction and prevent discomfort while talking to the other user [38]. For virtual conferences, having many senders and receivers, the additional demand of being able to adapt the video content for a large number of users while still maintaining low latency arises [25]. One-to-many applications, such as a virtual classroom in the context of education or a virtual concert in the context of entertainment, represent a hybrid category, wherein, instead of multiple participants generating content, a single peer serves as the content source, thereby simplifying the system design required to ensure low latency.…”

The Shenzhen-Hong Kong Dialectics

“…The confluence of video and the recently booming 3D representation technology embraces a new video paradigm, i.e., the volumetric video (VV). Different from traditional 2D video that has mature codecs based on frames and pixels, volumetric video is still in its infant stage with various representation formats, such as point cloud [11,34], voxel [33], mesh [34], and even neural representations [23]. Volumetric video is envisioned as a fundamental service that is able to facilitate various new applications such as extended reality (XR) and Metaverse, empowering entertainment [20], healthcare [7], and education [2], etc.…”