A Multimodal Emotional Human–Robot Interaction Architecture for Social Robots Engaged in Bidirectional Communication

Hong, Alexander; Lunscher, Nolan; Hu, Tianhao; Tsuboi, Yuma; Zhang, Xinyi; Alves, Silas Franco dos Reis; Nejat, Goldie; Benhabib, B.

doi:10.1109/tcyb.2020.2974688

Cited by 71 publications

(24 citation statements)

References 97 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Robots can use one or more sensors (camera, microphone, pressure, and physiological) to recognize an emotion. Some models are considered multimodal emotional, such as the model proposed by [ 108 ], a novel multimodal emotional architecture designed to promote natural and engaging bidirectional emotional communication between social robots and humans. Emotional communication is detected using a combination of modalities such as body language and vocal intonation.…”

Section: Resultsmentioning

confidence: 99%

Affective Communication for Socially Assistive Robots (SARs) for Children with Autism Spectrum Disorder: A Systematic Review

Cano

González

Gil

et al. 2021

Sensors

View full text Add to dashboard Cite

Research on affective communication for socially assistive robots has been conducted to enable physical robots to perceive, express, and respond emotionally. However, the use of affective computing in social robots has been limited, especially when social robots are designed for children, and especially those with autism spectrum disorder (ASD). Social robots are based on cognitive-affective models, which allow them to communicate with people following social behaviors and rules. However, interactions between a child and a robot may change or be different compared to those with an adult or when the child has an emotional deficit. In this study, we systematically reviewed studies related to computational models of emotions for children with ASD. We used the Scopus, WoS, Springer, and IEEE-Xplore databases to answer different research questions related to the definition, interaction, and design of computational models supported by theoretical psychology approaches from 1997 to 2021. Our review found 46 articles; not all the studies considered children or those with ASD.

show abstract

Section: Resultsmentioning

confidence: 99%

Affective Communication for Socially Assistive Robots (SARs) for Children with Autism Spectrum Disorder: A Systematic Review

Cano

González

Gil

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…Multimodal ML is a research field with one of its earliest application in the field of Audio-Visual Speech Recognition (AVSR) [8], to contemporary fields of emotion recognition [9], [10] and tracking multiple agents [11] or Anomalies in traffic [12]. Like sensor data fusion, in the sense that it uses multiple sensor modalities, Multimodal ML uses multiple ML algorithms, either in parallel or in series, to improve a framework [13].…”

Section: Related Workmentioning

confidence: 99%

A Multimodal Data Processing System for LiDAR-Based Human Activity Recognition

Roche

De-Silva

Hook

et al. 2022

IEEE Trans. Cybern.

View full text Add to dashboard Cite

Increasingly, the task of detecting and recognising the actions of a human has been delegated to some form of neural network processing camera or wearable sensor data. Due to the degree to which the camera can be affected by lighting and wearable sensors scantiness, neither one modality can capture the required data to perform the task confidently. That being the case, range sensors like Light Detection and Ranging (LiDAR) can complement the process to perceive the environment more robustly. Most recently, researchers have been exploring ways to apply Convolutional Neural Networks to 3-Dimensional data. These methods typically rely on a single modality and cannot draw on information from complementing sensor streams to improve accuracy. This paper proposes a framework to tackle human activity recognition by leveraging the benefits of sensor fusion and multimodal machine learning. Given both RGB and point cloud data, our method describes the activities being performed by subjects using regions with a convolutional neural network (R-CNN) and a 3-Dimensional modified Fisher vector network. Evaluated on a custom captured multimodal dataset demonstrates that the model outputs remarkably accurate human activity classification (90%). Furthermore, this framework can be used for sports analytics, understanding social behaviour, surveillance, and perhaps most notably by Autonomous Vehicles (AV) to datadriven decision-making policies in urban areas and indoor environments.

show abstract

“…Social robots interact with users socially and it is often zerocontact between the robot and the user, which challenges the applicability of current safeguards focussing solely on pHRI . Recent advances in social and industrial robots allow robots to read social cues, anticipate behaviour, and even predict emotions (Cid et al, 2014;Hong et al, 2020), therefore giving them sophisticated social abilities (Henschel et al, 2020). How individuals interact with robots in a given context and for some time can have adverse short and longterm effects on their well-being and overall behaviour.…”

Section: Psychological Componentmentioning

confidence: 99%

Redefining Safety in Light of Human-Robot Interaction: A Critical Review of Current Standards and Regulations

et al. 2021

View full text Add to dashboard Cite

Policymakers need to consider the impacts that robots and artificial intelligence (AI) technologies have on humans beyond physical safety. Traditionally, the definition of safety has been interpreted to exclusively apply to risks that have a physical impact on persons’ safety, such as, among others, mechanical or chemical risks. However, the current understanding is that the integration of AI in cyber-physical systems such as robots, thus increasing interconnectivity with several devices and cloud services, and influencing the growing human-robot interaction challenges how safety is currently conceptualised rather narrowly. Thus, to address safety comprehensively, AI demands a broader understanding of safety, extending beyond physical interaction, but covering aspects such as cybersecurity, and mental health. Moreover, the expanding use of machine learning techniques will more frequently demand evolving safety mechanisms to safeguard the substantial modifications taking place over time as robots embed more AI features. In this sense, our contribution brings forward the different dimensions of the concept of safety, including interaction (physical and social), psychosocial, cybersecurity, temporal, and societal. These dimensions aim to help policy and standard makers redefine the concept of safety in light of robots and AI’s increasing capabilities, including human-robot interactions, cybersecurity, and machine learning.

show abstract

A Multimodal Emotional Human–Robot Interaction Architecture for Social Robots Engaged in Bidirectional Communication

Cited by 71 publications

References 97 publications

Affective Communication for Socially Assistive Robots (SARs) for Children with Autism Spectrum Disorder: A Systematic Review

Affective Communication for Socially Assistive Robots (SARs) for Children with Autism Spectrum Disorder: A Systematic Review

A Multimodal Data Processing System for LiDAR-Based Human Activity Recognition

Redefining Safety in Light of Human-Robot Interaction: A Critical Review of Current Standards and Regulations

Contact Info

Product

Resources

About