On Using Social Signals to Enable Flexible Error-Aware HRI

Stiber, Maia; Taylor, Russell H.; Huang, Chien-Ming

doi:10.1145/3568162.3576990

Cited by 16 publications

(7 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While their binary failure detection model achieves good performance (up to 77% accuracy), the authors report the difficulties of transferability of failure detection models into new contexts, as well as in detecting different failure types. More recently, Stiber et al [8] leveraged facial expressions of humans interacting with a robot for error detection models across three HRI tasks: collaborative assembly, collaborative cooking, and programming by demonstration. Their work reinforces the idea that human reactions to failure are complex and vary according to the context of the failure, but facial expressions show promise as inputs for error detection models.…”

Section: Leveraging Social Cues In Hrimentioning

confidence: 99%

“…We note that in most existing literature (as reviewed by [2,3]), research studies were not accompanied by the publication or release of raw datasets of people's responses to robot errors (Stiber et al [8]'s groundbreaking study featured a dataset of facial reactions to robotic errors, but includes only the facial action units associated with user reactions). This lack of available datasets is likely due to the fact that the studies to date feature data collected from in-situ interactions; from a study perspective, this data has the most ecological validity, but the models trained on this data are not intended to work in any other context.…”

Section: Relevant Datasetsmentioning

confidence: 99%

“…Future work could involve expanding BADNet by including multimodal sources of data. For human social cues, one could include vector representations of features such as facial expressions [8,31], prosodic/audio data [32], and body position or body motion [33]. Contextual data inputs may also be used, such as additional sensor data from the environment, to allow the robot to use information about its surroundings to detect what or when something failed.…”

Section: Future Workmentioning

confidence: 99%

“…Prior work has studied how humans react to robot failures [4][5][6], revealing complex and multimodal responses. More recently, research in the field of Human-Robot Interaction (HRI) is investigating the potential of these social cues to inform robots that an error has occurred [7,8]. However, these works study settings where the human directly interacts with the robot under very controlled conditions that frequently involve explicitly designed failures performed by a limited number of robots.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

The Bystander Affect Detection (BAD) Dataset for Failure Detection in HRI

Bremers¹,

Parreira²,

Fang³

et al. 2023

Preprint

View full text Add to dashboard Cite

For a robot to repair its own error, it must first know it has made a mistake. One way that people detect errors is from the implicit reactions from bystanders -their confusion, smirks, or giggles clue us in that something unexpected occurred. To enable robots to detect and act on bystander responses to task failures, we developed a novel method to elicit bystander responses to human and robot errors. Using 46 different stimulus videos featuring a variety of human and machine task failures, we collected a total of 2452 webcam videos of human reactions from 54 participants. To test the viability of the collected data, we used the bystander reaction dataset as input to a deep-learning model, BADNet, to predict failure occurrence. We tested different data labeling methods and learned how they affect model performance, achieving precisions above 90%. We discuss strategies to model bystander reactions and predict failure and how this approach can be used in real-world robotic deployments to detect errors and improve robot performance. As part of this work, we also contribute with the "Bystander Affect Detection" (BAD) dataset of bystander reactions, supporting the development of better prediction models.

show abstract

Section: Leveraging Social Cues In Hrimentioning

confidence: 99%

Section: Relevant Datasetsmentioning

confidence: 99%

Section: Future Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

The Bystander Affect Detection (BAD) Dataset for Failure Detection in HRI

Bremers¹,

Parreira²,

Fang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…We expand the existing line of research on predicting impressions of robot performance from nonverbal human behavior to dynamic scenarios involving robot navigation. Prior work has often considered stationary tasks, like physical assembly at a desk [9] or robot photography [4], in laboratory environments. We instead explore the potential of using observations of body motion, gaze, and facial expressions to predict a human's impressions of robot performance while a robot guides them to a destination in a crowded environment.…”

Section: Introductionmentioning

confidence: 99%

Recent Advances in Mobile Robot Localization in Complex Scenarios

Zhang

Liu

et al. 2023

Lecture Notes in Electrical Engineering

View full text Add to dashboard Cite

Human impressions of robot performance are often measured through surveys. As a more scalable and cost-effective alternative, we study the possibility of predicting people's impressions of robot behavior using non-verbal behavioral cues and machine learning techniques. To this end, we first contribute the SEAN TOGETHER Dataset consisting of observations of an interaction between a person and a mobile robot in a Virtual Reality simulation, together with impressions of robot performance provided by users on a 5-point scale. Second, we contribute analyses of how well humans and supervised learning techniques can predict perceived robot performance based on different combinations of observation types (e.g., facial, spatial, and map features). Our results show that facial expressions alone provide useful information about human impressions of robot performance; but in the navigation scenarios we tested, spatial features are the most critical piece of information for this inference task. Also, when evaluating results as binary classification (rather than multiclass classification), the F1-Score of human predictions and machine learning models more than doubles, showing that both are better at telling the directionality of robot performance than predicting exact performance ratings. Based on our findings, we provide guidelines for implementing these predictions models in real-world navigation scenarios.

show abstract