Development of a Repository of Virtual 3D Conversational Gestures and Expressions

Capturing Conversational Gestures for Embodied Conversational Agents Using an Optimized Kaneda–Lucas–Tomasi Tracker and Denavit–Hartenberg-Based Kinematic Model

“…However, synthetic gestures still lack believability. To cope with the challenge of addressing the liveliness, diversity, and consistency of synthetic gestures adequately, we propose to exploit gesture tracking and 3D reconstruction to deliver a system capable of recording gestures expressed in the video during face-to-face conversation automatically, and storing them as gesture prototypes of the so-called “motor skills” [ 20 ]. We built the concept based on the following assumption: ECAs with diverse sets of resources, which preserve the dynamics and complexity of human movement, will be more successful in their attempt to mimic human-like conversational behavior.…”

“…However, they may decrease believability significantly when the internal fluidity (dynamics) is changed due to different intent or prosodic context. To increase the versatility of the gestural morphology and to decrease the complexity of designing the movements a conversational agent can reproduce, we propose to build a 3D corpus of gesture prototypes [ 20 ]. For the modeling environment, we exploited the DAZ3D studio, which simplifies the CAD controls, and, through a vast repertoire of conversational resources, minimizes the human animator’s need for ‘artistic’ skills.…”

“…With our main motivation in mind, i.e., to capture conversational expressions from monocular video as similar to the original as possible, and by preserving the ‘prosody of movement’, i.e., fluidity and dynamics, we designed a novel system, which consists of a Kanade–Lucas–Tomasi tracker (KLT) [ 45 ] to track the observed body parts based on optical flow, and a Denavit–Hartenberg-based kinematic model [ 46 ] to reconstruct tracked features as 3D templates and store them as part of the EVA’s [ 20 ] motor skills repository. However, as with any image processing algorithm, mismatches in either tracking or reconstruction will always appear.…”