Automatic replication of teleoperator head movements and facial expressions on a humanoid robot

Ondras, Jan; Çeliktutan, Oya; Sarıyanidi, Evangelos; Güneş, Hatice

doi:10.1109/roman.2017.8172386

Cited by 8 publications

(3 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unlike relatively more flexible virtual agents, robots inherently have less expressiveness due to the physical limitations of their actuators, in the aspects of shape, degree of freedom, smoothness, speed, and so on. Researchers have been working on the imitation of the head movements of humans for tele-operated robots [29][30][31]. These works do not synthesize robots' head movements in an autonomous manner but aim to replicate the head movements of the robot's human operator in real-time.…”

Section: Related Workmentioning

confidence: 99%

Generation of Head Movements of a Robot Using Multimodal Features of Peer Participants in Group Discussion Conversation

Huang

Kimura

Kuwabara

et al. 2020

MTI

View full text Add to dashboard Cite

In recent years, companies have been seeking communication skills from their employees. Increasingly more companies have adopted group discussions during their recruitment process to evaluate the applicants’ communication skills. However, the opportunity to improve communication skills in group discussions is limited because of the lack of partners. To solve this issue as a long-term goal, the aim of this study is to build an autonomous robot that can participate in group discussions, so that its users can repeatedly practice with it. This robot, therefore, has to perform humanlike behaviors with which the users can interact. In this study, the focus was on the generation of two of these behaviors regarding the head of the robot. One is directing its attention to either of the following targets: the other participants or the materials placed on the table. The second is to determine the timings of the robot’s nods. These generation models are considered in three situations: when the robot is speaking, when the robot is listening, and when no participant including the robot is speaking. The research question is: whether these behaviors can be generated end-to-end from and only from the features of peer participants. This work is based on a data corpus containing 2.5 h of the discussion sessions of 10 four-person groups. Multimodal features, including the attention of other participants, voice prosody, head movements, and speech turns extracted from the corpus, were used to train support vector machine models for the generation of the two behaviors. The performances of the generation models of attentional focus were in an F-measure range between 0.4 and 0.6. The nodding model had an accuracy of approximately 0.65. Both experiments were conducted in the setting of leave-one-subject-out cross validation. To measure the perceived naturalness of the generated behaviors, a subject experiment was conducted. In the experiment, the proposed models were compared. They were based on a data-driven method with two baselines: (1) a simple statistical model based on behavior frequency and (2) raw experimental data. The evaluation was based on the observation of video clips, in which one of the subjects was replaced by a robot performing head movements in the above-mentioned three conditions. The experimental results showed that there was no significant difference from original human behaviors in the data corpus and proved the effectiveness of the proposed models.

show abstract

Section: Related Workmentioning

confidence: 99%

Generation of Head Movements of a Robot Using Multimodal Features of Peer Participants in Group Discussion Conversation

Huang

Kimura

Kuwabara

et al. 2020

MTI

View full text Add to dashboard Cite

show abstract

“…This feature vector was then z-normalised, and was fed into the pose regression model as in the offline mode. Differently from the offline mode, smoothing was performed using the Kalman filter, as it is commonly applied in real-time motion synthesis systems [46,51]. The pose vectors were finally sent to the robot, and the adaptive sleep time was applied as in the offline mode.…”

Section: Synthesis Phasementioning

confidence: 99%

Audio-Driven Robot Upper-Body Motion Synthesis

Ondras

Çeliktutan

Bremner

et al. 2021

IEEE Trans. Cybern.

Self Cite

View full text Add to dashboard Cite

Body language is an important aspect of human communication, which an effective human-robot interaction interface should mimic well. Human beings exchange information and convey their thoughts and feelings through gaze, facial expressions, body language and tone of voice along with spoken words, and infer 65% of the meaning of the communicated messages from these nonverbal cues. Modern robotic platforms are however limited in their ability to automatically generate behaviours that align with their speech. In this paper, we develop a neural network based system that takes audio from a user as an input and generates upper-body gestures including head, hand and torso movements of the user on a humanoid robot, namely, Softbank Robotics' Pepper. Our system was evaluated quantitatively as well as qualitatively using web-surveys when driven by natural speech and synthetic speech. We compare the impact of generic and personspecific neural network models on the quality of synthesised movements. We further investigate the relationships between quantitative and qualitative evaluations and examine how the speaker's personality traits affect the synthesised movements.

show abstract

“…We therefore extracted four types of features, namely, shape, appearance, differential-appearance and differential-shape features. The details of this system are described by Ondras et al [34].…”

Section: (B) the Teachmeeq System For Automatic Expression Prediction (I) The Teachmeeq Systemmentioning

confidence: 99%

Live human–robot interactive public demonstrations with automatic emotion and personality prediction

Güneş

Çeliktutan

Sarıyanidi

2019

Phil. Trans. R. Soc. B

Self Cite

View full text Add to dashboard Cite

Communication with humans is a multi-faceted phenomenon where the emotions, personality and non-verbal behaviours, as well as the verbal behaviours, play a significant role, and human–robot interaction (HRI) technologies should respect this complexity to achieve efficient and seamless communication. In this paper, we describe the design and execution of five public demonstrations made with two HRI systems that aimed at automatically sensing and analysing human participants’ non-verbal behaviour and predicting their facial action units, facial expressions and personality in real time while they interacted with a small humanoid robot. We describe an overview of the challenges faced together with the lessons learned from those demonstrations in order to better inform the science and engineering fields to design and build better robots with more purposeful interaction capabilities. This article is part of the theme issue ‘From social brains to social robots: applying neurocognitive insights to human–robot interaction’.

show abstract

Automatic replication of teleoperator head movements and facial expressions on a humanoid robot

Cited by 8 publications

References 26 publications

Generation of Head Movements of a Robot Using Multimodal Features of Peer Participants in Group Discussion Conversation

Generation of Head Movements of a Robot Using Multimodal Features of Peer Participants in Group Discussion Conversation

Audio-Driven Robot Upper-Body Motion Synthesis

Live human–robot interactive public demonstrations with automatic emotion and personality prediction

Contact Info

Product

Resources

About