Project starline

Abstract

We present a real-time bidirectional communication system that lets two people, separated by distance, experience a face-to-face conversation as if they were copresent. It is the first telepresence system that is demonstrably better than 2D videoconferencing, as measured using participant ratings (e.g., presence, attentiveness, reaction-gauging, engagement), meeting recall, and observed nonverbal behaviors (e.g., head nods, eyebrow movements). This milestone is reached by maximizing audiovisual fidelity and the sense of copresence in all design elements, including physical layout, lighting, face tracking, multi-view capture, microphone array, multi-stream compression, loudspeaker output, and lenticular display. Our system achieves key 3D audiovisual cues (stereopsis, motion parallax, and spatialized audio) and enables the full range of communication cues (eye contact, hand gestures, and body language), yet does not require special glasses or body-worn microphones/headphones. The system consists of a head-tracked autostereoscopic display, high-resolution 3D capture and rendering subsystems, and network transmission using compressed color and depth video streams. Other contributions include a novel image-based geometry fusion algorithm, free-space dereverberation, and talker localization.

Drivable Volumetric Avatars using Texel-Aligned Features

“…Drivable Telepresence Systems. Drivable telepresence systems can be divided into two categories; Non-parametric solutions are holistically modeling a scene and faithfully transmit observed signals as it is [Lawrence et al 2021;Martin-Brualla et al 2018;Orts-Escolano et al 2016]. The other solution is to leverage a category-specific parametric model for driving [Bagautdinov et al 2021;Lombardi et al 2018;Xiang et al 2021].…”

“…Non-parametric telepresence systems, including Holoportation [Orts-Escolano et al 2016], and Project Starline [Lawrence et al 2021], leverage classical 3D reconstruction techniques and imagespace post-processing for real-time rendering of the scene. While these approaches do not require per-user training, its fidelity is bounded by the input signals.…”

“…An alternative path to building telepresence systems showing promise, is to rely on model-free methods which combine classical geometry reconstruction methods with image-space processing, either with ad-hoc image fusion [Lawrence et al 2021] or neural re-rendering [Martin-Brualla et al 2018;Shysheya et al 2019]. Such methods are able to better exploit available inputs, but usually require highly specialized hardware for high-fidelity real-time sensing and would be limited in handling occlusions and incomplete data, either exhibiting limited quality rendering of novel viewpoints [Martin-Brualla et al 2018] or requiring high viewpoint coverage by using multiple highly-specialized sensors [Lawrence et al 2021]. Moreover, due to the challenges in real-time sensing, artifacts may occur around occlusion boundaries and body parts with limited resolution, like fingers or hair.…”

RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touch

“…Studies [11,49] have shown that to achieve this goal, the system should faithfully capture facial expressions and body gestures and support natural eye contact between remote meeting participants. Recent advances [39,82] in 3D telepresence and 3D video communication make it possible to create convincing visual illusion that remote meeting participants are in the same room. While these advances are encouraging, much work remains.…”

“…This rendering task is highly non-trivial. Existing image-based rendering solutions such as [39,82] can realistically render the participants hands when they are positioned away from the screen (above a distance threshold) and thus visible by the RGBD cameras that provide the color and depth video feeds to the rendering algorithms. However, these rendering solutions stop working when the hand is too close to the screen (below the distance threshold and thus invisible to the RGBD cameras).…”