Social Behavior Modeling Based on Incremental Discrete Hidden Markov Models

Mihoub, Alaeddine; Bailly, Gérard; Wolf, Christian

doi:10.1007/978-3-319-02714-2_15

Cited by 8 publications

(5 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Figueroa-Angulo et al [16] trained a compound hidden Markov model to recognize human activity with RGB-D skeleton data of humans for a service robot. For the related problem of face-to-face conversation, conversation estimation has been demonstrated using visual tracking alone [51][52][53] or combined RGB-D sensing to analysing and generating multimodal behaviour [49]. Mihoub et al's approach for social behaviour modelling and generation is based on incremental discrete hidden Markov models.…”

Section: Related Workmentioning

confidence: 99%

Automatically Classifying User Engagement for Dynamic Multi-party Human–Robot Interaction

Foster

Gaschler

Giuliani

2017

Int J of Soc Robotics

View full text Add to dashboard Cite

A robot agent designed to engage in real-world human-robot joint action must be able to understand the social states of the human users it interacts with in order to behave appropriately. In particular, in a dynamic public space, a crucial task for the robot is to determine the needs and intentions of all of the people in the scene, so that it only interacts with people who intend to interact with it. We address the task of estimating the engagement state of customers for a robot bartender based on the data from audiovisual sensors. We begin with an offline experiment using hidden Markov models, confirming that the sensor data contains the information necessary to estimate user state. We then present two strategies for online state estimation: a rule-based classifier based on observed human behaviour in real bars, and a set of supervised classifiers trained on a labelled corpus. These strategies are compared in offline cross-validation, in an online user study, and through validation against a separate test corpus. These studies show that while the trained classifiers are best in a cross-validation setting, the rule-based classifier performs best with novel data; however, all classiThis article integrates and extends the work described in the following conference papers: [17,20,23,24 Bristol Robotics Laboratory, University of the West of England, Bristol, UK fiers also change their estimate too frequently for practical use. To address this issue, we present a final classifier based on Conditional Random Fields: this model has comparable performance on the test data, with increased stability. In summary, though, the rule-based classifier shows competitive performance with the trained classifiers, suggesting that for this task, such a simple model could actually be a preferred option, providing useful online performance while avoiding the implementation and data-scarcity issues involved in using machine learning for this task.

show abstract

Section: Related Workmentioning

confidence: 99%

Automatically Classifying User Engagement for Dynamic Multi-party Human–Robot Interaction

Foster

Gaschler

Giuliani

2017

Int J of Soc Robotics

View full text Add to dashboard Cite

show abstract

“…The coordination between eyes, head and possibly other segments of the body depends on numerous factors such as the actual physical disposition of the interlocutors, their social roles and status (see the Multidimensional Dimensional Scaling analysis performed on gaze models in [37]) as well as the context of the interaction. We will see how a multimodal gaze control model can be biased by these physical and social settings.…”

Section: Discussionmentioning

confidence: 99%

On the Benefit of Independent Control of Head and Eye Movements of a Social Robot for Multiparty Human-Robot Interaction

Haefflinger

Elisei

Gerber

et al. 2023

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

The human gaze direction is the sum of the head and eye movements. The coordination of these two segments has been studied and models of the contribution of head movement to the gaze of virtual agents or robots have been proposed. However, these coordination models are mostly not trained nor evaluated in an interaction context, and may underestimate the social functions of gaze. Indeed, after analyzing human behavior in a three-party conversation dataset, we show that the contribution of the head to the gaze varies depending on whether the speaker is addressing two interlocutors or one of them: the conversational regime actually impacts the head/eyes coordination. We therefore propose an evaluation of different coordination policies in a social interaction context, using a Furhat robot to replay the human multimodal behavior from our data record. The verbal content and gaze targets are the same, but the robot uses four different head and eye coordination policies. (1) Furhat's default gaze control, whose eyes move faster and start before the head, but finally aligns both segments. (2) the robot head is fixed and only the eyes move. (3) the eyes are fixed and only the head moves. (4) Human-like control where the robot mimics the head movements of the human dataset, which naturally exploits independent eye and head control. Using an online crowdsourced test, we show that the human-like policy, which uses decoupled head and eye movements, is perceived significantly more natural than the others.

show abstract

“…We build also interlocutor-dependent (ID) models [26,64]: a set of 10 ID models is built using data from each dyad. Mirroring the training of II models, each ID model is thus trained on one interaction and tested on the 9 remaining ones.…”

Section: Analyzing Speaker-dependent Modelsmentioning

confidence: 99%

Learning multimodal behavioral models for face-to-face social interaction

Mihoub

Bailly

Wolf

et al. 2015

J Multimodal User Interfaces

Self Cite

View full text Add to dashboard Cite

International audienceThe aim of this paper is to model multimodal perception-action loops of human behavior in face-to-face interactions. The long-term goal of this research is to give artificial agents social skills to engage believable interactions with human interlocutors. To this end, we propose trainable behavioral models that generate optimal actions given others’ perceived actions and joint goals. We first compare sequential models - in particular Discrete Hidden Markov Models (DHMMs) - with standard classifiers (SVMs and Decision Trees). We propose a modification of the initialization of the DHMMs in order to better capture the recurrent structure of the sensory-motor states. We show that the explicit state duration modeling by Hidden Semi Markov Models (HSMMs) improves prediction performance. We applied these models to parallel speech and gaze data collected from interacting dyads. The challenge was to predict the gaze of one subject given the gaze of the interlocutor and the voice activity of both. For both HMMs and HSMMs the Short-Time Viterbi concept is used for incremental decoding and generation. For the proposed models we evaluated objectively many properties in order to go beyond pure classification performance. Results show that while Incremental Discrete HMMs (IDHMMs) were more efficient than classic classifiers, the Incremental Discrete HSMMs (IDHSMMs) give best performance. This result emphasizes the relevance of state duration modeling

show abstract

Social Behavior Modeling Based on Incremental Discrete Hidden Markov Models

Cited by 8 publications

References 25 publications

Automatically Classifying User Engagement for Dynamic Multi-party Human–Robot Interaction

Automatically Classifying User Engagement for Dynamic Multi-party Human–Robot Interaction

On the Benefit of Independent Control of Head and Eye Movements of a Social Robot for Multiparty Human-Robot Interaction

Learning multimodal behavioral models for face-to-face social interaction

Contact Info

Product

Resources

About