Capturing Conversational Gestures for Embodied Conversational Agents Using an Optimized Kaneda–Lucas–Tomasi Tracker and Denavit–Hartenberg-Based Kinematic Model

Abstract

In order to recreate viable and human-like conversational responses, the artificial entity, i.e., an embodied conversational agent, must express correlated speech (verbal) and gestures (non-verbal) responses in spoken social interaction. Most of the existing frameworks focus on intent planning and behavior planning. The realization, however, is left to a limited set of static 3D representations of conversational expressions. In addition to functional and semantic synchrony between verbal and non-verbal signals, the final believability of the displayed expression is sculpted by the physical realization of non-verbal expressions. A major challenge of most conversational systems capable of reproducing gestures is the diversity in expressiveness. In this paper, we propose a method for capturing gestures automatically from videos and transforming them into 3D representations stored as part of the conversational agent’s repository of motor skills. The main advantage of the proposed method is ensuring the naturalness of the embodied conversational agent’s gestures, which results in a higher quality of human-computer interaction. The method is based on a Kanade–Lucas–Tomasi tracker, a Savitzky–Golay filter, a Denavit–Hartenberg-based kinematic model and the EVA framework. Furthermore, we designed an objective method based on cosine similarity instead of a subjective evaluation of synthesized movement. The proposed method resulted in a 96% similarity.

Computer Vision in Human Analysis: From Face and Body to Clothes

“…The task of 3D human pose estimation is addressed in [ 3 ], in which the authors propose the use of bidirectional gated recurrent units to predict the global motion sequence from the local pose sequence. Gestures are investigated in [ 4 ], where a method for capturing gestures automatically from videos and transforming them into stored 3D representations is proposed. In [ 5 ], the authors exploit body joints to predict action progress.…”

Direct and Inverse Cinematic Resolution Calculation using Denavit-Hartenberg Algorithm and Geometric Method for Robot with 5 DOF

“…Saraf et al, 2021 [22] utilized MATLAB and implemented the D-H algorithm's parameters for the modelling process of a 6 DOF robotic arm driven by a hydraulic system. Mocnik et al, 2022 [23] propose a method which transforms videos captures of gestures and stores them as 3D models in the repository of a conversational agent to be used for its motor skills. Ciorneu et al, 2019 [24] offered a method applied to spatial mechanisms in which the kinematics are studied and solved.…”

“…The fundamental trigonometric identity was used to replace sine theta 2 in it and obtain the tangent ratio in ( 20), ( 21) and (22). Finally, theta 2 was found using arctan2 as shown in (23).…”

LiDAR-Based Maintenance of a Safe Distance between a Human and a Robot Arm

“…Another possibility, often employed, is to represent a robot arm by a mathematical model. When a kinematic model is used, the geometry of the robot arm is established from its state, provided by its controller [19][20][21][22]. On the other hand, more complex dynamic models represent the motion of the robot arm, as well as the forces acting on it and on the human during the collision [23,24].…”