A multi-modal intelligent user interface for supervisory control of unmanned platforms

Taylor, Glenn; Frederiksen, R. A.; Crossman, Jacob; Quist, Michael C.; Theisen, Patrick

doi:10.1109/cts.2012.6261037

Cited by 13 publications

(5 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A display that integrates situation awareness and robot control is an obvious future step. Our ongoing smart interaction device (SID) research [7] will complement this work by enabling natural multi-modal interactions with the UV team, potentially obviating the need for a physical control interface at all in many situations. In future work we anticipate integrating SAGE with these and other similar technologies developed by third parties.…”

Section: Use In the Magic International Robot Competition: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

A hands-off, multi-robot display for communicating situation awareness to operators

Crossman¹,

Marinier²,

Olson

2012

2012 International Conference on Collaboration Technologies and Systems (CTS)

Self Cite

View full text Add to dashboard Cite

Abstract-In this paper we introduce work toward this goal of enabling an operator to effectively maintain situation awareness over a team of heterogeneous robots. Most existing operator control units (OCUs) are designed for tele-operation of a single robot and require the constant attention of the operator. As the role of robots increase in important fields such as bomb disposal; intelligence, surveillance, and reconnaissance (ISR); and search and rescue it becomes critical to improve this ratio by enabling multiple robots to be monitored and controlled by a single operator. Toward this goal we have developed the Situations, Actions, Goals, and Environment (SAGE) interface, which utilizes numerous techniques for maintaining operator situation awareness by aiding the operator's perception, comprehension, and projection when controlling a team of ground robots. Techniques that we developed include dynamic view control, event detection and prioritization, uncertainty display, view-invariant markers to make state information globally visible, and animations to avoid jarring distractions. SAGE's interface was automated making it completely hands off, thus the operator could save interaction time for critical tasks such robot tasking. SAGE was deployed by Team Michigan as part of its OCU suite in the 2010 MultiAutonomous Ground-robotic International Challenge (MAGIC). Team Michigan won this competition in part because of its innovative OCU designs, including SAGE, that enabled two operators to control 14 robots simultaneously -a 1-to-7 operator to robot ratio, higher than any other team.

show abstract

Section: Use In the Magic International Robot Competition: Resultsmentioning

confidence: 99%

“…Thus one thread of research (including some of our own work [7]) has focused on providing information via multiple modalities such as voice or force feedback.…”

Section: Prior Workmentioning

confidence: 99%

A hands-off, multi-robot display for communicating situation awareness to operators

Crossman¹,

Marinier²,

Olson

2012

2012 International Conference on Collaboration Technologies and Systems (CTS)

Self Cite

View full text Add to dashboard Cite

show abstract

“…This interaction, as well as any feedback to the user, is managed through the Dialogue Manager component in SID Core. (See [22] for more details. )…”

Section: Figure 1: High Level Architecture For the Smart Interaction mentioning

confidence: 96%

Natural interaction for unmanned systems

Taylor¹,

Purman²,

Schermerhorn³

et al. 2015

SPIE Proceedings

Self Cite

View full text Add to dashboard Cite

Military unmanned systems today are typically controlled by two methods: tele-operation or menu-based, search-andclick interfaces. Both approaches require the operator's constant vigilance: tele-operation requires constant input to drive the vehicle inch by inch; a menu-based interface requires eyes on the screen in order to search through alternatives and select the right menu item. In both cases, operators spend most of their time and attention driving and minding the unmanned systems rather than on being a warfighter. With these approaches, the platform and interface become more of a burden than a benefit. The availability of inexpensive sensor systems in products such as Microsoft Kinect™ or Nintendo Wii™ has resulted in new ways of interacting with computing systems, but new sensors alone are not enough. Developing useful and usable human-system interfaces requires understanding users and interaction in context: not just what new sensors afford in terms of interaction, but how users want to interact with these systems, for what purpose, and how sensors might enable those interactions. Additionally, the system needs to reliably make sense of the user's inputs in context, translate that interpretation into commands for the unmanned system, and give feedback to the user. In this paper, we describe an example natural interface for unmanned systems, called the Smart Interaction Device (SID), which enables natural two-way interaction with unmanned systems including the use of speech, sketch, and gestures. We present a few example applications SID to different types of unmanned systems and different kinds of interactions.

show abstract

“…Multimodal interfaces have also proven effective with robots, such as in Correa et al (2010) where a combination of sketching and voice commands were used to supervise an autonomous forklift. This concept is expanded in Taylor et al (2012) where sketching and a dialogue between the robot and user are utilized to develop context toward the command. An intuitive supervisory control system can be utilized for an unmanned system.…”

Section: Related Workmentioning

confidence: 99%

Command and Control of a Large Scale Swarm Using Natural Human Interfaces

Williamson¹,

Taranta²,

Moolenaar³

et al. 2023

View full text Add to dashboard Cite

While swarm robotics contain varying levels of autonomy, it is still important for human operators to command and control the swarm before and during mission operations. As such, humanin- the-loop interfaces must be developed that are intuitive to the operator and able to relay accurate commands to flight and ground robots in a large-scale swarm network. Furthermore, the tremendous amount of information gathered by the swarm must be sorted and presented back to the user in a way that reduces cognitive load so that they may determine the relevancy of the information and its effect on the mission. In this paper, we present sketch-based and augmented reality interfaces that both allow for command and control (C2) of unmanned ground and flight vehicles and for processing of data that the swarm has gathered in the field. These interfaces were tested in two field experiments with multiple aerial and ground robots deployed on a mission. Finally, we discuss the results of the field experiments, lessons learned, and areas for future work.

show abstract

A multi-modal intelligent user interface for supervisory control of unmanned platforms

Cited by 13 publications

References 19 publications

A hands-off, multi-robot display for communicating situation awareness to operators

A hands-off, multi-robot display for communicating situation awareness to operators

Natural interaction for unmanned systems

Command and Control of a Large Scale Swarm Using Natural Human Interfaces

Contact Info

Product

Resources

About