Defining Next-Generation Multi-Modal Communication in Human Robot Interaction

Abstract

With teleoperation being the contemporary standard for Human Robot Interaction (HRI), research into multi-modal communication (MMC) has focused on development of advanced Operator Control Units (OCU) supporting control of one or more robots. However, with advances being made to improve the perception, intelligence, and mobility of robots, a need exists to revolutionize the ways in which Soldiers interact with robotic team members. Within this future vision, mixed-initiative Soldier-Robot (SR) teams will work collaboratively sharing information back-and-forth in a fluid natural manner using combinations of communication methods. Therefore, new definitions are required to focus research efforts to support next-generation MMC. After a thorough survey of the literature and a scientific workshop on the topic, this paper aims to operationally define MMC, Explicit Communication, and Implicit Communication to encompass the shifting paradigm of HRI from a controller/controlled relationship to a cooperative team mate relationship. This paper presents the results from a survey of the literature and a scientific workshop that inform proposed definitions for multi-modal, explicit, and implicit communication. An illustrative scenario vignette provides context and specific examples of each communication type. Finally, future research efforts are summarized.

Improving Teamwork Competencies in Human-Machine Teams: Perspectives From Team Science

“…While machines currently possess the ability to detect certain implicit cues; e.g., via the recognition of facial expressions (Picard et al, 2001), they are limited in their ability to detect contextual cues. For example, because coordination involves a complex and varying presentation of implicit communication cues (Lackey et al, 2011), it is difficult and expensive to support machine cue perception on a human level. Detection, interpretation, and reasoning about these cues from a human perspective (Baker et al, 2011) is imperative to ensure effective coordination.…”

“…In team science, adaptation or adaptability as a teamwork competency is examined more broadly and refers to the adjustment of strategies and behaviors in response to changes in the team's circumstances (Driskell et al, 2018). In considering adaptation through this lens, machines are capable of detecting changes in the internal team and external environments (Lackey et al, 2011), allowing them to engage both adaptive and adaptable mechanisms as designed. They may also detect some underlying causes of changing environments through common sense reasoning (Morgenstern et al, 2016), though this capability remains limited by the datasets used to train common sense (Hao, 2020).…”

Multimodal Interface for Human–Robot Collaboration

“…The type of such a communication can be explicit or implicit [31]. Explicit communication includes intentional interaction stimuli through the modality, such as the use of a specific trigger word or pointing to an object.…”

The Application and Extension of the Human-Animal Team Model to Better Understand Human-Robot Interaction: Recommendations for Further Research

“…Others have noted that alternative, non-verbal communications provide a potential avenue for investigating bi-directional communication between humans and robots (Bockelman-Morrow & Fiore, 2012). Signaling and non-verbal commands commonly used in human-animal teams demonstrate the utility of such communication without any reliance on explicit language driven approaches (Lackey, Barber, Reinerman, Badler, & Hudson, 2011;Phillips et al, 2016). For instance, in some contexts (such as military operations), simplistic forms of communication (e.g., gestures, verbal commands) like those employed in human-animal teams, can be beneficial in supporting teaming that can generalize to a variety of missions, especially those in which team members need to communication quickly, over long distances, or in instances of stealth operations.…”