Gaze Complements Control Input for Goal Prediction During Assisted Teleoperation

Abstract

Shared control systems can make complex robot teleoperation tasks easier for users. These systems predict the user's goal, determine the motion required for the robot to reach that goal, and combine that motion with the user's input. Goal prediction is generally based on the user's control input (e.g., the joystick signal). In this paper, we show that this prediction method is especially effective when users follow standard noisily optimal behavior models. In tasks with input constraints like modal control, however, this effectiveness no longer holds, so additional sources for goal prediction can improve assistance. We implement a novel shared control system that combines natural eye gaze with joystick input to predict people's goals online, and we evaluate our system in a real-world, COVID-safe user study. We fnd that modal control reduces the effciency of assistance according to our model, and when gaze provides a prediction earlier in the task, the system's performance improves. However, gaze on its own is unreliable and assistance using only gaze performs poorly. We conclude that control input and natural gaze serve different and complementary roles in goal prediction, and using them together leads to improved assistance. * * sum a with the user command u directly: a exec = a + u.

Using Eye Gaze to Train an Adaptive Myoelectric Interface

“…4). This is consistent with the findings in Madduri et al [42] and Aronson et al [3], where the former showed that their task-trained adaptive interface improved tracking performance within minutes of training and the latter showed that natural gaze assisted in goal predicting for device control. However, as expected, the conventional mouse interface enables significantly better tracking than both myoelectric interfaces, underscoring the challenges in designing personalized myoelectric interfaces with high-dimensional inputs.…”

“…Several studies have also integrated people’s natural gaze in human-robot interaction and have proved its advantages in anticipating user selections or intentions, thus improving system efficiency and task accuracy [1, 3, 11, 30, 63]. Aronson et al [3] used natural gaze to predict users’ task goals and then used the predictions to improve a learning algorithm for manual (joystick)-based robot manipulation. Similarly, this study also leverages gaze to approximate the user’s desired actions.…”

“…Conventional gaze-controlled interfaces generally require users to focus their gaze to produce input, which limits natural eye movements during everyday tasks and can lead to fatigue [34]. Natural eye behavior, in contrast, offers insights into user’s movement goals [35, 46], which has the potential to augment human-computer systems with indirect signals [3]. Therefore, we hypothesized that eye movements can replace task-based data in training adaptive myoelectric interfaces by assuming that natural gaze serves as a proxy for user intent and task goals.…”

Gaze-Based Shared Autonomy Framework With Real-Time Action Primitive Recognition for Robot Manipulators

“…Aronson and Admoni proposed an intent inference method for gaze-based shared autonomy systems [25]. A Partially Observable Markov Decision Process model used joystick and eye tracker signals in order to update probability distributions for candidate target objects.…”

AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System

“…Compared to gripper-based manipulators, teleoper-ating dexterous hand-arm systems poses unprecedented challenges and often requires specialized apparatus that comes with high costs and setup efforts, such as Virtual Reality (VR) devices [4,17,15], wearable gloves [29,30], handheld controller [45,46,20], haptic sensors [12,23,50,53], or motion capture trackers [65]. Fortunately, recent developments in vision-based teleoperation [2,24,16,26,42,27,21,22,3] have provided a low-cost and more generalizable alternative for teleoperating dexterous robot systems.…”