Modeling Human Response to Robot Errors for Timely Error Detection

Stiber, Maia; Taylor, Russell H.; Huang, Chien-Ming

doi:10.1109/iros47612.2022.9981726

Cited by 16 publications

(9 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Notably, a body of work has explored the potential of human nonverbal reactions to detect robot failures in HRI [5,52,70,71,100,131,139]. We will describe these works in more detail in the following sections.…”

Section: Key Related Literature Reviewsmentioning

confidence: 99%

“…They found that lexical features (tone, affect, positive and negative emotion) are highly significant in predicting task failure, and multi-modal streams of information perform better for failure classification. Facial expressions were leveraged in Stiber et al [131] to detect and localize robot errors in HRI. The authors built a two-stage model: 1) weighted binary classification and 2) filtering through a sliding window.…”

Section: Supervised Learningmentioning

confidence: 99%

See 1 more Smart Citation

Using Social Cues to Recognize Task Failures for HRI: A Review of Current Research and Future Directions

Bremers¹,

Pabst²,

Parreira³

et al. 2023

Preprint

View full text Add to dashboard Cite

Robots that carry out tasks and interact in complex environments will inevitably commit errors. Error detection is thus an important ability for robots to master, to work in an efficient and productive way. People leverage social cues from others around them to recognize and repair their own mistakes. With advances in computing and AI, it is increasingly possible for robots to achieve a similar error detection capability. In this work, we review current literature around the topic of how social cues can be used to recognize task failures for human-robot interaction (HRI). This literature review unites insights from behavioral science, human-robot interaction, and machine learning, to focus on three areas: 1) social cues for error detection (from behavioral science), 2) recognizing task failures in robots (from HRI), and 3) approaches for autonomous detection of HRI task failures based on social cues (from machine learning). We propose a taxonomy of error detection based on self-awareness and social feedback. Finally, we leave recommendations for HRI researchers and practitioners interested in developing robots that detect (physical) task errors using social cues from bystanders.

show abstract

Section: Key Related Literature Reviewsmentioning

confidence: 99%

Section: Supervised Learningmentioning

confidence: 99%

Using Social Cues to Recognize Task Failures for HRI: A Review of Current Research and Future Directions

Bremers¹,

Pabst²,

Parreira³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…For each data point i, we collect a total of n = 88 measurements M i = [m 0 , ..., m n ] (Table 3), as well as a label l i . Feature extraction for classification is performed through a sliding window of predefined length (L = 3.5s or 7s) [42,53] and hop length h = 0.5s. For every window (i.e.…”

Section: Extracted Featuresmentioning

confidence: 99%

Multi-modal Affect Detection Using Thermal Imaging in a Gamified Exercise

Mohamed

Özgür

Lemaignan

et al. 2023

Preprint

View full text Add to dashboard Cite

Affect recognition, or the ability to detect and interpret emotional states, has the potential to be a valuable tool in the field of healthcare. In particular, it can be useful in gamified therapy, which involves using gaming techniques to motivate and keep the engagement of patients in therapeutic activities. This study aims to examine the effectiveness of a tangible PacMan game in inducing different emotions in participants and the accuracy of machine learning models using thermal imaging and action unit data for affect classification. A self-report survey and three machine learning models were used to assess emotions including frustration, boredom, and enjoyment in participants during different phases of the game. The results showed that the game was successful in inducing significant differences in enjoyment and frustration, but only enjoyment and frustration were significantly different from boredom. The machine learning models showed that the multimodal approach with the combination of thermal imaging and action units had the highest accuracy of 77% for emotion classification in the 7-second window, while thermal imaging had the lowest standard deviation among participants. The results suggest that thermal imaging and action units can be effective in detecting affective states and might have the potential to be used in healthcare applications, such as gamified therapy, as a promising non-intrusive method for detecting internal states.

show abstract

“…Often, these social cues can communicate one's internal states with respect to the surrounding context. The range of social responses to error is widely studied in the behavioral science literature: facial expressions [7,14,15], physiological responses [16,17], gestures and vocal expressions [5,18], among others [2,3].…”

Section: Social Cues In Reactions To Failurementioning

confidence: 99%

“…Prior work has studied how humans react to robot failures [4][5][6], revealing complex and multimodal responses. More recently, research in the field of Human-Robot Interaction (HRI) is investigating the potential of these social cues to inform robots that an error has occurred [7,8]. However, these works study settings where the human directly interacts with the robot under very controlled conditions that frequently involve explicitly designed failures performed by a limited number of robots.…”

Section: Introductionmentioning

confidence: 99%

The Bystander Affect Detection (BAD) Dataset for Failure Detection in HRI

Bremers¹,

Parreira²,

Fang³

et al. 2023

Preprint

View full text Add to dashboard Cite

For a robot to repair its own error, it must first know it has made a mistake. One way that people detect errors is from the implicit reactions from bystanders -their confusion, smirks, or giggles clue us in that something unexpected occurred. To enable robots to detect and act on bystander responses to task failures, we developed a novel method to elicit bystander responses to human and robot errors. Using 46 different stimulus videos featuring a variety of human and machine task failures, we collected a total of 2452 webcam videos of human reactions from 54 participants. To test the viability of the collected data, we used the bystander reaction dataset as input to a deep-learning model, BADNet, to predict failure occurrence. We tested different data labeling methods and learned how they affect model performance, achieving precisions above 90%. We discuss strategies to model bystander reactions and predict failure and how this approach can be used in real-world robotic deployments to detect errors and improve robot performance. As part of this work, we also contribute with the "Bystander Affect Detection" (BAD) dataset of bystander reactions, supporting the development of better prediction models.

show abstract

Modeling Human Response to Robot Errors for Timely Error Detection

Cited by 16 publications

References 24 publications

Using Social Cues to Recognize Task Failures for HRI: A Review of Current Research and Future Directions

Using Social Cues to Recognize Task Failures for HRI: A Review of Current Research and Future Directions

Multi-modal Affect Detection Using Thermal Imaging in a Gamified Exercise

The Bystander Affect Detection (BAD) Dataset for Failure Detection in HRI

Contact Info

Product

Resources

About