Integrated Intelligence for Human-Robot Teams

Oh, Jean; Howard, Thomas M.; Walter, Matthew R.; Barber, Daniel; Zhu, Menglong; Park, Sang‐Don; Suppé, Arne; Navarro-Serment, Luis E.; Duvallet, Felix; Boularias, Abdeslam; Romero, Oscar J.; Vinokurov, Jerry; Keegan, Terence; Dean, Robert; Lennon, Craig; Bodt, Barry A.; Childers, Marshal A.; Shi, Jianbo; Daniilidis, Kostas; Roy, Nicholas; Lebière, Christian; Hebert, Martial; Stentz, Anthony

doi:10.1007/978-3-319-50115-4_28

Cited by 20 publications

(19 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The sequential execution of the navigate, search, and observe actions constitutes a complete mission, examples of which are presented with more detail in recent work by Oh et al 3 . Here we present the methodology and results of the assessments of each action as tested separately.…”

Section: Observementioning

confidence: 98%

“…These experiments tested semantic navigation and perception, human-robot interaction, door detection, and pedestrian detection and tracking. The results of the tests of human-robot interaction are presented in Hill et al 2 , and the results of the complete runs, are presented Oh et al 3 . Here, we present results from the experiments testing semantic navigation and perception, door detection, and pedestrian detection and tracking.…”

Section: Introductionmentioning

confidence: 97%

See 1 more Smart Citation

RCTA capstone assessment

et al. 2015

Self Cite

View full text Add to dashboard Cite

The Army Research Laboratory's Robotics Collaborative Technology Alliance (RCTA) is a program intended to change robots from tools that soldiers use into teammates with which soldiers can work. This requires the integration of fundamental and applied research in perception, artificial intelligence, and human-robot interaction. In October of 2014, the RCTA assessed progress towards integrating this research. This assessment was designed to evaluate the robot's performance when it used new capabilities to perform selected aspects of a mission. The assessed capabilities included the ability of the robot to: navigate semantically outdoors with respect to structures and landmarks, identify doors in the facades of buildings, and identify and track persons emerging from those doors. We present details of the mission-based vignettes that constituted the assessment, and evaluations of the robot's performance in these vignettes.

show abstract

Section: Observementioning

confidence: 98%

Section: Introductionmentioning

confidence: 97%

RCTA capstone assessment

et al. 2015

Self Cite

View full text Add to dashboard Cite

show abstract

“…Earlier work in this area includes that of Duvallet et al (2013), which learns to follow navigational instructions in unknown environments based upon human demonstrations, as well as recent work on language-based visual navigation in novel environments (Anderson et al, 2018;Mei et al, 2016a). More closely related to our framework are methods that leverage metric and semantic information implicit or explicit in the command to learn a distribution over world models that facilitates natural language understanding in a priori unknown environments (Duvallet et al, 2014;Oh et al, 2016;Walter et al, 2014b). We address a different element of ''partial observability'' by inferring the state of Fig.…”

Section: Related Workmentioning

confidence: 99%

Multimodal estimation and communication of latent semantic knowledge for robust execution of robot instructions

Arkin

Park

Roy

et al. 2020

The International Journal of Robotics Research

Self Cite

View full text Add to dashboard Cite

The goal of this article is to enable robots to perform robust task execution following human instructions in partially observable environments. A robot’s ability to interpret and execute commands is fundamentally tied to its semantic world knowledge. Commonly, robots use exteroceptive sensors, such as cameras or LiDAR, to detect entities in the workspace and infer their visual properties and spatial relationships. However, semantic world properties are often visually imperceptible. We posit the use of non-exteroceptive modalities including physical proprioception, factual descriptions, and domain knowledge as mechanisms for inferring semantic properties of objects. We introduce a probabilistic model that fuses linguistic knowledge with visual and haptic observations into a cumulative belief over latent world attributes to infer the meaning of instructions and execute the instructed tasks in a manner robust to erroneous, noisy, or contradictory evidence. In addition, we provide a method that allows the robot to communicate knowledge dissonance back to the human as a means of correcting errors in the operator’s world model. Finally, we propose an efficient framework that anticipates possible linguistic interactions and infers the associated groundings for the current world state, thereby bootstrapping both language understanding and generation. We present experiments on manipulators for tasks that require inference over partially observed semantic properties, and evaluate our framework’s ability to exploit expressed information and knowledge bases to facilitate convergence, and generate statements to correct declared facts that were observed to be inconsistent with the robot’s estimate of object properties.

show abstract

“…Chung et al [11] use HDCG on ground vehicles to implement navigation commands and demonstrate performance improvements over G 3 in terms of running time, factor evaluations, and correctness. Oh et al [12] integrate HDCG with their navigating robot system, The overall pipeline of our approach highlighting the NLP parsing module and the motion planner. Above the dashed line (from left to right): Dynamic Grounding Graphs (DGG) with latent parameters that are used to parse and interpret the natural language commands, generation of optimization-based planning formulation with appropriate constraints and parameters using our mapping algorithm.…”

Section: A Natural Language Processingmentioning

confidence: 99%

“…Most prior methods that combine NLP and motion planning have focused on understanding natural language instructions to compute robot motion for simple environments and constraints. Most of these methods are limited to navigation applications [12], [11], [6] or simple settings [7], or they are not evaluated on real robots [10]. Nyga et al [26], [27], [28], [29] use probabilistic relation models based on knowledge bases to understand natural language commands that describe visual attributes of objects.…”

Section: Benefits and Comparisonsmentioning

confidence: 99%

Efficient Generation of Motion Plans from Attribute-Based Natural Language Instructions Using Dynamic Constraint Mapping

Park

Jia²,

Bansal³

et al. 2019

2019 International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

We present an algorithm for combining natural language processing (NLP) and fast robot motion planning to automatically generate robot movements. Our formulation uses a novel concept called Dynamic Constraint Mapping to transform complex, attribute-based natural language instructions into appropriate cost functions and parametric constraints for optimization-based motion planning. We generate a factor graph from natural language instructions called the Dynamic Grounding Graph (DGG), which takes latent parameters into account. The coefficients of this factor graph are learned based on conditional random fields (CRFs) and are used to dynamically generate the constraints for motion planning. We map the cost function directly to the motion parameters of the planner and compute smooth trajectories in dynamic scenes. We highlight the performance of our approach in a simulated environment and via a human interacting with a 7-DOF Fetch robot using intricate language commands including negation, orientation specification, and distance constraints.

show abstract

Integrated Intelligence for Human-Robot Teams

Cited by 20 publications

References 16 publications

RCTA capstone assessment

RCTA capstone assessment

Multimodal estimation and communication of latent semantic knowledge for robust execution of robot instructions

Efficient Generation of Motion Plans from Attribute-Based Natural Language Instructions Using Dynamic Constraint Mapping

Contact Info

Product

Resources

About