Combining dynamic head pose–gaze mapping with the robot conversational state for attention recognition in human–robot interactions

Abstract

The ability to recognize the Visual Focus of Attention (VFOA, i.e. what or whom a person is looking at) of people is important for robots or conversational agents interacting with multiple people, since it plays a key role in turn-taking, engagement or intention monitoring. As eye gaze estimation is often impossible to achieve, most systems currently rely on head pose as an approximation, creating ambiguities since the same head pose can be used to look at different VFOA targets. To address this challenge, we propose a dynamic Bayesian model for the VFOA recognition from head pose, where we make two main contributions. First, taking inspiration from behavioral models describing the relationships between the body, head and gaze orientations involved in gaze shifts, we propose novel gaze models that dynamically and more accurately predict the expected head orientation used for looking in a given gaze target direction. This is a neglected aspect of previous works but essential for recognition. Secondly, we propose to exploit the robot conversational state (when he speaks, objects to which he refers) as context to set appropriate priors on candidate VFOA targets and reduce the inherent VFOA ambiguities. Experiments on a public dataset where the humanoid robot NAO plays the role of an art guide and quiz master demonstrate the benefit of the two contributions.

Tracking Gaze and Visual Focus of Attention of People Involved in Social Interaction

“…Moreover, [31] uses cross-modal information, namely the speaker identity based on the audio track (one of the participants or the robot) as well as the identity of the object of interest. We also note that [31] reports mean FRR values obtained over all the test recordings, instead of an FRR value for each recording. Table IV summarizes a comparison between the average FRR obtained with our method, with [26], and with [31].…”

“…This can be explained by the fact that social interactions, even in different contexts, share a lot of characteristics. We compared our method with three other methods, based on HMMs [26], on input-output HMMs [31], and on a geometric model [4]. The interest of these methods (including ours) resides in the fact that eye detection, unlike many existing gaze estimation methods, is not needed.…”

“…This work was extended to deal with more complex scenarios, such as participants interacting with a robot [27], [31]. An input-output HMM is proposed in [31] to enable to model the following contextual information: participants tend to look to the speaker, to the robot, or to an object which is referred to by the speaker or by the robot. The results of [31] show that this improves the performance of VFOA estimation.…”

“…An input-output HMM is proposed in [31] to enable to model the following contextual information: participants tend to look to the speaker, to the robot, or to an object which is referred to by the speaker or by the robot. The results of [31] show that this improves the performance of VFOA estimation. Nevertheless, this method requires additional information, such as speaker identification or speech recognition.…”

Entropy-weighted feature-fusion method for head-pose estimation

“…Experimental results have revealed considerable differences between statistical methods and non-statistical measurements [23][24][25][26]. The former mainly focus on appearance-based measurements, whereas the latter usually consider geometric relationship cues, such as the deviation of the nose from the mid-line and the deviation between the new head pose and the original state.…”