Director: A User Interface Designed for Robot Operation with Shared Autonomy

Abstract

Operating a high degree of freedom mobile manipulator, such as a humanoid, in a field scenario requires constant situational awareness, capable perception modules, and effective mechanisms for interactive motion planning and control. A well‐designed operator interface presents the operator with enough context to quickly carry out a mission and the flexibility to handle unforeseen operating scenarios robustly. By contrast, an unintuitive user interface can increase the risk of catastrophic operator error by overwhelming the user with unnecessary information. With these principles in mind, we present the philosophy and design decisions behind Director—the open‐source user interface developed by Team MIT to pilot the Atlas robot in the DARPA Robotics Challenge (DRC). At the heart of Director is an integrated task execution system that specifies sequences of actions needed to achieve a substantive task, such as drilling a wall or climbing a staircase. These task sequences, developed a priori, make online queries to automated perception and planning algorithms with outputs that can be reviewed by the operator and executed by our whole‐body controller. Our use of Director at the DRC resulted in efficient high‐level task operation while being fully competitive with approaches focusing on teleoperation by highly trained operators. We discuss the primary interface elements that comprise Director, and we provide an analysis of its successful use at the DRC.

Autonomous car driving by a humanoid robot

“…Most of the existing approaches have achieved this goal by relying on teleoperation. [16][17][18][19][20][21][22] Supervisory steering and gas commands are sent to the robot to drive the car in Karumanchi et al 23 ; DeDonato and colleagues 24 propose a hybrid solution, with teleoperated steering and autonomous speed control. The velocity of the car, estimated with stereo cameras, is fed back to a proportional integral (PI) controller, whereas light imaging, detection, and ranging (LIDAR), IMU, and visual odometry data support the operator during the steering procedures.…”

“…Note, in fact, that even if the robot ankle moves in a small range, the car speed changes significantly. The noise on the ankle command, as well as the initial peak, are due to the derivative term of the gas pedal control (22). However, the signal is smoothed by the task-based QP control (see the dashed black line, i.e., the signal reconstructed by encoder readings), preventing jerky motion of the robot foot.…”

Robust shared autonomy for mobile manipulation with continuous scene monitoring

“…The DIRECTOR interface and system architecture used to control the ATLAS robot, developed by Team MIT, is described in [8] and [9] respectively. It features a shared autonomy system backed by interactive, assisted perception, and trajectory optimization-based motion planning [10] where task sequences can be created from high-level motion primitives and constraints.…”

“…Our shared autonomy builds on the task execution system described in [8] which fills a sequence of task primitives with details acquired through operator-assisted perception online. The operator can review, pause, and amend the execution at any time, and the autonomous mode can be resumed immediately after a phase of manual operation.…”

“…We expand on [8] to extend the task sequence system with automatic review and approval of planned motions through continuous scene monitoring and reasoning to reduce the amount of human intervention required, and only fall back to the operator when absolutely necessary. In order to achieve this, a task tree with task dependencies is defined that is automatically expanded to synthesize sequences comprised of high-level action primitives in response to perception input and changes in the environment.…”