Gaze correction for home video conferencing

Abstract

Figure 1: Top: frames recorded during video-conferencing. Note the lack of eye contact because of the disparity between the locations of the participant and the camera. Bottom: real-time gaze correction with the proposed algorithm. AbstractEffective communication using current video conferencing systems is severely hindered by the lack of eye contact caused by the disparity between the locations of the subject and the camera. While this problem has been partially solved for high-end expensive video conferencing systems, it has not been convincingly solved for consumer-level setups. We present a gaze correction approach based on a single Kinect sensor that preserves both the integrity and expressiveness of the face as well as the fidelity of the scene as a whole, producing nearly artifact-free imagery. Our method is suitable for mainstream home video conferencing: it uses inexpensive consumer hardware, achieves real-time performance and requires just a simple and short setup. Our approach is based on the observation that for our application it is sufficient to synthesize only the corrected face. Thus we render a gaze-corrected 3D model of the scene and, with the aid of a face tracker, transfer the gaze-corrected facial portion in a seamless manner onto the original image.

Extrinsic Camera Calibration with Minimal Configuration Using Cornea Model and Equidistance Constraint

“…From these results, we confirmed that our method works properly in real environments since the differences in the extrinsic parameters estimated by each method are small. While this precision is not enough for eye gaze tracking, it is acceptable for applications that do not need high precision, such as gaze correction [12] using a display and attached web camera system.…”

“…They assume that the user's gaze points represent his/her interests in the contents on display. As another example, Kuster et al [12] propose a gaze correction method with a display-camera setup for home video conferences. For these applications, they have to know the relative posture and position of the camera against the display.…”

An experimental approach to study the physiology of natural social interactions

“…The absence of direct eye contact in video conferencing is critical to investigate natural social interactions, for which mutual gaze is a central element eliciting a sense of presence and an engagement (Senju & Johnson, 2009). Recently, technical solutions to this limitation have been proposed, based on online video correction (Kuster et al, 2012). An alternative option is the use of a device based on a semi-transparent mirror system.…”

Eye contact over video

“…More recently this approach has been extended with the use of depth sensors for better results, such as in [12]. Very visually promising results are also presented in [4] where a Kinect is used to rotate a facial texture in a video frame making it appear as if the person is looking straight at the camera. Notably [5] demonstrates the capturing of an entire environment into a 3D model, and subsequent rendering of virtual viewpoints in real time.…”