Deep compositional robotic planners that follow natural language commands

Kuo, Yen-Ling; Katz, Boris; Barbu, Andrei

doi:10.1109/icra40945.2020.9197464

Cited by 18 publications

(12 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Robot navigation [10] and path planning from natural language instructions have been widely investigated in the field of robotics [11]- [13]. Such a task was recently formalized adopting data-driven methods [4] with the release of the R2R dataset.…”

Section: Related Workmentioning

confidence: 99%

CrossMap Transformer: A Crossmodal Masked Path Transformer Using Double Back-Translation for Vision-and-Language Navigation

Magassouba

Sugiura

Kawai

2021

Preprint

View full text Add to dashboard Cite

Navigation guided by natural language instructions is particularly suitable for Domestic Service Robots that interacts naturally with users. This task involves the prediction of a sequence of actions that leads to a specified destination given a natural language navigation instruction. The task thus requires the understanding of instructions, such as "Walk out of the bathroom and wait on the stairs that are on the right". The Visual and Language Navigation remains challenging, notably because it requires the exploration of the environment and at the accurate following of a path specified by the instructions to model the relationship between language and vision. To address this, we propose the CrossMap Transformer network, which encodes the linguistic and visual features to sequentially generate a path. The CrossMap transformer is tied to a Transformer-based speaker that generates navigation instructions. The two networks share common latent features, for mutual enhancement through a double back translation model: Generated paths are translated into instructions while generated instructions are translated into path The experimental results show the benefits of our approach in terms of instruction understanding and instruction generation.

show abstract

Section: Related Workmentioning

confidence: 99%

CrossMap Transformer: A Crossmodal Masked Path Transformer Using Double Back-Translation for Vision-and-Language Navigation

Magassouba

Sugiura

Kawai

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…We use the intuition developed in Kuo et al (2020a) that models which are composed of sub-networks can disentangle the meaning of words in a sentence without direct supervision. In other words, the agent is never told what

is supposed to mean as opposed to

.…”

Section: Modelmentioning

confidence: 99%

Compositional RL Agents That Follow Language Commands in Temporal Logic

Kuo¹,

Katz²,

Barbu³

2021

Front. Robot. AI

Self Cite

View full text Add to dashboard Cite

We demonstrate how a reinforcement learning agent can use compositional recurrent neural networks to learn to carry out commands specified in linear temporal logic (LTL). Our approach takes as input an LTL formula, structures a deep network according to the parse of the formula, and determines satisfying actions. This compositional structure of the network enables zero-shot generalization to significantly more complex unseen formulas. We demonstrate this ability in multiple problem domains with both discrete and continuous state-action spaces. In a symbolic domain, the agent finds a sequence of letters that satisfy a specification. In a Minecraft-like environment, the agent finds a sequence of actions that conform to a formula. In the Fetch environment, the robot finds a sequence of arm configurations that move blocks on a table to fulfill the commands. While most prior work can learn to execute one formula reliably, we develop a novel form of multi-task learning for RL agents that allows them to learn from a diverse set of tasks and generalize to a new set of diverse tasks without any additional training. The compositional structures presented here are not specific to LTL, thus opening the path to RL agents that perform zero-shot generalization in other compositional domains.

show abstract

“…Similar to attention-based predictors [20,12], attention plays a key role in our model and guides the trajectory predictions. Although the scope of the attention module is gated by the linguistic descriptions [21][22][23], agents that don't co-occur in the same description don't need to be jointly attended to. Language guides attention which ultimately removes extraneous agents and improves performance.…”

Section: A Trajectory Prediction With Attention and Structuresmentioning

confidence: 99%

“…The input linguistic commands for planners mostly specify goals and referents, e.g., landmarks, for navigation or locating the objects to interact with. Some planners learn visual attention maps conditioned on language to predict a goal map [27], filter relevant objects [23], or estimate a visitation map [28] for a single agent. However, none of these planners considers language describing interactions with other agents and manners of performing an action, such as quickly or slowly.…”

Section: B Language and Planningmentioning

confidence: 99%

Trajectory Prediction with Linguistic Representations

Kuo¹,

Huang²,

Barbu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Language allows humans to build mental models that interpret what is happening around them resulting in more accurate long-term predictions. We present a novel trajectory prediction model that uses linguistic intermediate representations to forecast trajectories, and is trained using trajectory samples with partially-annotated captions. The model learns the meaning of each of the words without direct per-word supervision. At inference time, it generates a linguistic description of trajectories which captures maneuvers and interactions over an extended time interval. This generated description is used to refine predictions of the trajectories of multiple agents. We train and validate our model on the Argoverse dataset, and demonstrate improved accuracy results in trajectory prediction. In addition, our model is more interpretable: it presents part of its reasoning in plain language as captions, which can aid model development and can aid in building confidence in the model before deploying it.

show abstract

Deep compositional robotic planners that follow natural language commands

Cited by 18 publications

References 18 publications

CrossMap Transformer: A Crossmodal Masked Path Transformer Using Double Back-Translation for Vision-and-Language Navigation

CrossMap Transformer: A Crossmodal Masked Path Transformer Using Double Back-Translation for Vision-and-Language Navigation

Compositional RL Agents That Follow Language Commands in Temporal Logic

Trajectory Prediction with Linguistic Representations

Contact Info

Product

Resources

About