Attention allocation for human multi-robot control: Cognitive analysis based on behavior data and hidden states

Abstract

Human multi-robot interaction exploits both the human operator's high-level decision-making skills and the robotic agents' vigorous computing and motion abilities. While controlling multi-robot teams, an operator's attention must constantly shift between individual robots to maintain sufficient situation awareness. To conserve an operator's attentional resources, a robot with self-reflect capability on its abnormal status can help an operator focus her attention on emergent tasks rather than unneeded routine checks. With the proposing self-reflect aids, the human-robot interaction becomes a queuing framework, where the robots act as the clients to request for interaction and an operator acts as the server to respond these job requests. This paper examined two types of queuing schemes, the self-paced Open-queue identifying all robots' normal/abnormal conditions, whereas the forced-paced shortest-job-first (SJF) queue showing a single robot's request at one time by following the SJF approach. As a robot may miscarry its experienced failures in various situations, the effects of imperfect automation were also investigated in this paper. The results suggest that the SJF attentional scheduling approach can provide stable performance in both primary (locate potential targets) and secondary (resolve robots' failures) tasks, regardless of the system's reliability levels. However, the conventional results (e.g., number of targets marked) only present little information about users' underlying cognitive strategies and may fail to reflect the user's true intent. As understanding users' intentions is critical to providing appropriate cognitive aids to enhance task performance, a Hidden Markov Model (HMM) is used to examine operators' underlying cognitive intent and identify the unobservable cognitive states. The HMM results demonstrate fundamental differences among the queuing mechanisms and reliability conditions. The findings suggest that HMM can be helpful in investigating the use of human cognitive resources under multitasking environments.

Multirobot Confidence and Behavior Modeling: An Evaluation of Semiautonomous Task Performance and Efficiency

“…While our model of confidence could drive overt feedback to the operator or be applied only to internal processes of the robot, the implementation presented here is directed at minimally intrusive adjustment of physical behavior to mitigate the challenges of human interaction with multiple mobile robots. The online application distinguishes this work from others which estimated operator attention offline [52] or used human eye gaze for training [128].…”

“…General approaches to address operator overload due to multitasking include redesigning tasks and interfaces to reduce demands, training operators to develop automaticity, improve attention management, and automating tasks and task management [51]. Research toward interaction with multiple semiautonomous robots includes task switching and operator attention allocation [52][53][54], such as identifying where an operator should focus and influencing the operator's behavior accordingly via visual cues in a graphical user interface [55]. Other work includes determining which aspects of a given task are most suitable for automation [16], measuring and influencing operator trust in team autonomy [19], using intelligent agents to help human operators manage a team of multiple robots [13], and augmented reality interfaces to integrate information from multiple sources and project it into a view of the real world using a common frame of reference [35,56,57].…”

“…Their main recommendation is that a robot learns specific policies based on examples. Chien, Lin, and Lee [52] proposed a hidden Markov model (HMM) to examine operator intent, and performed offline HMM analysis of multirobot interaction queuing mechanisms. Several groups [59][60][61] (including our own) have used eye-gaze tracking to determine the user intent for zooming the camera.…”

Augmented visualization cues on primary flight display facilitating pilot's monitoring performance

“…Pilots often must deal with time critical situations, it is important that pilots can distribute their attention effectively between the raw data and its relevant modes, as failures to manage a high-priority task in a timely manner could lead to potentially disastrous consequences (Bybee et al, 2011). Therefore, it is important to apply cognitive assistance to support pilot's attention resources on the flight deck (Chien et al, 2018).…”

Internet of Green Things with autonomous wireless wheel robots against green houses and farms

“…Multi-agent systems are used for implementing HEMSs to illustrate the communication of agents among devices for energy sources in sequence. [43][44][45][46][47][48][49][50] Presently, no interacting low-cost and highly autonomous robots are found in the green houses. In particular, sensors are placed and data manipulation has been done manually or with the help of web interfaces.…”