An Intelligence Architecture for Grounded Language Communication with Field Robots

Abstract

For humans and robots to collaborate effectively as teammates in unstructured environments, robots must be able to construct semantically rich models of the environment, communicate efficiently with teammates, and perform sequences of tasks robustly with minimal human intervention, as direct human guidance may be infrequent and/or intermittent. Contemporary architectures for human-robot interaction often rely on engineered human-interface devices or structured languages that require extensive prior training and inherently limit the kinds of information that humans and robots can communicate. Natural language, particularly when situated with a visual representation of the robot’s environment, allows humans and robots to exchange information about abstract goals, specific actions, and/or properties of the environment quickly and effectively. In addition, it serves as a mechanism to resolve inconsistencies in the mental models of the environment across the human-robot team. This article details a novel intelligence architecture that exploits a centralized representation of the environment to perform complex tasks in unstructured environments. The centralized environment model is informed by a visual perception pipeline, declarative knowledge, deliberate interactive estimation, and a multimodal interface. The language pipeline also exploits proactive symbol grounding to resolve uncertainty in ambiguous statements through inverse semantics. A series of experiments on three different, unmanned ground vehicles demonstrates the utility of this architecture through its robust ability to perform language-guided spatial navigation, mobile manipulation, and bidirectional communication with human operators. Experimental results give examples of component-level behaviors and overall system performance that guide a discussion on observed performance and opportunities for future innovation.

LINGO-Space: Language-Conditioned Incremental Grounding for Space

“…Composite instructions: Composite linguistic instructions often introduce structural ambiguity. Researchers often use parsing as a solution, breaking down composite expressions using hand-crafted or grammatical rules (Tellex et al 2011;Howard et al 2022). Recently, Zhao, Lee, and Hsu (2023) have introduced the grounding method, PARAGON, in which its neural parsing module extracts object-centric relations.…”

Natural Language-Guided Semantic Navigation Using Scene Graph

“…In robotics, natural language grounding (NLG) refers to the ability of a robot to understand linguistic instructions by establishing a connection between phrases or words in human language and the physical world. Researchers have introduced a wide variety of human instruction grounding approaches [4], [5], [6], [7], [8]. Classic methods involve identifying the structure and meaning of linguistic instructions [9].…”