Bidirectional Multi-modal Signs of Checking Human-Robot Engagement and Interaction

Abstract

The anthropomorphization of human-robot interactions is a fundamental aspect of the design of social robotics applications. This article describes how an interaction model based on multimodal signs like visual, auditory, tactile, proxemic, and others can improve the communication between humans and robots. We have examined and appropriately filtered all the robot sensory data needed to realize our interaction model. We have also paid a lot of attention to communication on the backchannel, making it both bidirectional and evident through auditory and visual signals. Our model, based on a task-level architecture, was integrated into an application called W@ICAR, which proved efficient and intuitive with people not interacting with the robot. It has been validated both from a functional and user experience point of view, showing positive results. Both the pragmatic and the hedonic estimators have shown how many users particularly appreciated the application. The model component has been implemented through Python scripts in the robot operating system environment.

Automatic Context-Aware Inference of Engagement in HMI: A Survey

“…The robot judges the user's engagement based on the head's position which indicates if the user is looking at the robot, at objects necessary for the collaboration or to other objects or to empty space. Similarly, [184] implemented a multimodal rule-based real-time robotic architecture that detects user engagement based on movement detection (head and body tracking), face recognition, gaze direction, proxemic distance, and audio cues (sound direction localization, audio signal power). Other works that employed rule-based engagement inference methods include [55], [59], [58], [56], [157].…”

Recent advancements in multimodal human–robot interaction

“…The mobility signal of the robot and the client's cardiac signal are gathered and combined to provide multimodal data as the input node vector of the DL framework, which is utilized for the control system's model of HRI (Wang W. et al, 2022 ). Maniscalco et al ( 2022 ) evaluate and suitably filter all the robotic sensory data required to fulfill their interaction model, paying careful attention to backchannel interaction, making it bilateral and visible through audio and visual cues. Wang R. et al ( 2022 ) offer Husformer, a multimodal transformer architecture for multimodal human condition identification, suggesting the use of cross-modal transformers, which motivate one signal to strengthen itself by directly responding to latent relevancy disclosed in other signals.…”

Research on Guidance Methods of Digital Twin Robotic Arms Based on User Interaction Experience Quantification

“…However, the evolution of information technology, paired with the continuous upgrades of devices, has propelled virtual reality to offer a multimodal perception in human-computer interactions. Marrying digital twin interactivity with virtual reality environments aligns the interaction more congruently with user intuition [2], thereby enhancing the utility and efficacy of digital twin systems [3]. Specifically, within the domain of industrial robotic arms, remote operation and visualization of these digital twin robotic arms become feasible.…”