Achieving eye contact in a one-to-many 3D video teleconferencing system

Abstract

We present a set of algorithms and an associated display system capable of producing correctly rendered eye contact between a three-dimensionally transmitted remote participant and a group of observers in a 3D teleconferencing system. The participant's face is scanned in 3D at 30Hz and transmitted in real time to an autostereoscopic horizontal-parallax 3D display, displaying him or her over more than a 180° field of view observable to multiple observers. To render the geometry with correct perspective, we create a fast vertex shader based on a 6D lookup table for projecting 3D scene vertices to a range of subject angles, heights, and distances. We generalize the projection mathematics to arbitrarily shaped display surfaces, which allows us to employ a curved concave display surface to focus the high speed imagery to individual observers. To achieve two-way eye contact, we capture 2D video from a cross-polarized camera reflected to the position of the virtual participant's eyes, and display this 2D video feed on a large screen in front of the real participant, replicating the viewpoint of their virtual self. To achieve correct vertical perspective, we further leverage this image to track the position of each audience member's eyes, allowing the 3D display to render correct vertical perspective for each of the viewers around the device. The result is a one-to-many 3D teleconferencing system able to reproduce the effects of gaze, attention, and eye contact generally missing in traditional teleconferencing systems.

ConnectBoard: Enabling Genuine Eye Contact and Accurate Gaze in Remote Collaboration

“…The authors of [2] recently used a light field display in combination with a real-time 3-D capture system to deliver one-tomany eye contact. However, the light field display was only big enough to display one user's head, and the capture system is unable to recreate fine features like hair.…”

Advanced 360-Degree Integral-Floating Display Using a Hidden Point Removal Operator and a Hexagonal Lens Array

“…Horizontal parallax-only volumetric light field display allows the display of a 3D image in a 360-degree viewing zone by using high-speed projection and a rotating screen [12,13]. A 360-degree integral-floating display (IFD), which is a combination of integral imaging and a light field display, was proposed to improve the limited viewing angle of integral imaging display in the 360-degree horizontal direction [14].…”

Integral-floating Display with 360 Degree Horizontal Viewing Angle

“…Recently, a rotating-mirror-screen type volumetric light field display that projects 2D images to corresponding directions has been reported, demonstrating a 360 degree horizontal viewing angle which means displayed 3D image can be observed from any horizontal direction around the display system [15][16][17]. In an implementation, a high frame rate digital micro-mirror device (DMD) projects 288 different perspectives per round when the mirror screen rotates more than 900 rounds per minute (rpm) as shown in Fig.…”