Learning to Follow Language Instructions with Compositional Policies

Abstract

We propose a framework that learns to execute natural language instructions in an environment consisting of goalreaching tasks that share components of their task descriptions. Our approach leverages the compositionality of both value functions and language, with the aim of reducing the sample complexity of learning novel tasks. First, we train a reinforcement learning agent to learn value functions that can be subsequently composed through a Boolean algebra to solve novel tasks. Second, we fine-tune a seq2seq model pretrained on web-scale corpora to map language to logical expressions that specify the required value function compositions. Evaluating our agent in the BabyAI domain, we observe a decrease of 86% in the number of training steps needed to learn a second task after mastering a single task. Results from ablation studies further indicate that it is the combination of compositional value functions and language representations that allows the agent to quickly generalize to new tasks.* Work performed while an independent researcher.

Towards Zero-Shot Functional Compositionality of Language Models

“…Hence, Hupkes et al (2020) discusses this subject in depth by presenting a set of definitions and tests that is grounded on a vast amount of linguistic and philosophical literature, using probabilistic contextfree grammar datasets. Another very good example can also be found in visual recognition (Misra et al, 2017;Wang et al, 2019b;Naeem et al, 2021;Purushwalkam et al, 2019;Logeswaran et al, 2021;Cohen et al, 2021;Nayak et al, 2022). Here, if a model understands the meaning of the phrases "grey elephant" and "blue bottle", they test if it also generalizes to new vision-language concepts like "blue elephant".…”