A Survey of Reinforcement Learning Informed by Natural Language

Luketina, Jelena; Nardelli, Nantas; Farquhar, Gregory; Foerster, Jakob; Andreas, Jacob; Grefenstette, Edward; Whiteson, Shimon; Rocktäschel, Tim

doi:10.48550/arxiv.1906.03926

Cited by 24 publications

(29 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hermann et al, 2017;Kaplan et al, 2017), to target exploration (Goyal et al, 2019), or as an abstraction to structure hierarchical policies (Jiang et al, 2019). Luketina et al (2019) review a wide variety of recent uses of language in RL, and argue for further research. However, they do not even mention explanations.…”

Section: Related Work In Aimentioning

confidence: 99%

Tell me why! Explanations support learning relational and causal structure

Lampinen¹,

Roy²,

Dasgupta³

et al. 2021

Preprint

View full text Add to dashboard Cite

Explanations play a considerable role in human learning, especially in areas that remain major challenges for AI-forming abstractions, and learning about the relational and causal structure of the world. Here, we explore whether reinforcement learning agents might likewise benefit from explanations. We outline a family of relational tasks that involve selecting an object that is the odd one out in a set (i.e., unique along one of many possible feature dimensions). Odd-one-out tasks require agents to reason over multi-dimensional relationships among a set of objects. We show that agents do not learn these tasks well from reward alone, but achieve > 90% performance when they are also trained to generate language explaining object properties or why a choice is correct or incorrect. In further experiments, we show how predicting explanations enables agents to generalize appropriately from ambiguous, causally-confounded training, and even to meta-learn to perform experimental interventions to identify causal structure. We show that explanations help overcome the tendency of agents to fixate on simple features, and explore which aspects of explanations make them most beneficial. Our results suggest that learning from explanations is a powerful principle that could offer a promising path towards training more robust and general machine learning systems.Explanations-language that provides explicit information about the abstract, causal structure of the world-are central to human learning (Keil et al., 2000;Lombrozo, 2006). Explanations help solve the credit assignment problem, because they link a concrete situation to generalizable abstractions that can be used in the future (Lombrozo, 2006;Lombrozo and Carey, 2006). Thus explanations allow us to learn efficiently, from otherwise underspecified examples (Ahn et al., 1992). Human explanations selectively highlight generalizable causal factors and thereby improve our causal understanding (Lombrozo and Carey, 2006). Similarly, they help us to make comparisons and master re-

show abstract

Section: Related Work In Aimentioning

confidence: 99%

Tell me why! Explanations support learning relational and causal structure

Lampinen¹,

Roy²,

Dasgupta³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Despite the accomplishment of language models on NLP tasks, there have only been a few circumstances where these techniques have translated to the field of reinforcement learning. Luketina et al (2019) provides a survey of reinforcement learning methods which have been improved with the addition of natural language approaches. One such example is Kaplan, Sauer, and Sosa (2017), where natural language methods were used with deep reinforcement learning to play Atari games.…”

Section: Related Workmentioning

confidence: 99%

Reinforcement Learning with Information-Theoretic Actuation

Catt¹,

Hutter²,

Veness³

2021

Preprint

View full text Add to dashboard Cite

Reinforcement Learning formalises an embodied agent's interaction with the environment through observations, rewards and actions. But where do the actions come from? Actions are often considered to represent something external, such as the movement of a limb, a chess piece, or more generally, the output of an actuator. In this work we explore and formalize a contrasting view, namely that actions are best thought of as the output of a sequence of internal choices with respect to an action model. This view is particularly well-suited for leveraging the recent advances in large sequence models as prior knowledge for multi-task reinforcement learning problems. Our main contribution in this work is to show how to augment the standard MDP formalism with a sequential notion of internal action using information-theoretic techniques, and that this leads to self-consistent definitions of both internal and external action value functions.

show abstract

“…In this area as well the knowledge domain information can be used to improve the models and/or transform very difficult problems into more tractable ones; for example, by creating good initial conditions for the training algorithm, thus decreasing the number of required training examples and therefore providing a warm start for the reinforcement learning process [28]. Prior information has been successfully applied with remarkable benefits in various context well suited for reinforcement learning [18]. However, an important aspect has to be considered while injecting domain knowledge in this setting: uncertainty in the domain knowledge can greatly hinder the learning process if wrong decisions (caused by uncertain or incomplete information on the system state) are taken at the beginning of the learning phase [30].…”

Section: Related Areasmentioning

confidence: 99%

Improving Deep Learning Models via Constraint-Based Domain Knowledge: a Brief Survey

Borghesi,

Baldo,

Milano

2020

Preprint

View full text Add to dashboard Cite

Deep Learning (DL) models proved themselves to perform extremely well on a wide variety of learning tasks, as they can learn useful patterns from large data sets. However, purely data-driven models might struggle when very difficult functions need to be learned or when there is not enough available training data. Fortunately, in many domains prior information can be retrieved and used to boost the performance of DL models.This paper presents a first survey of the approaches devised to integrate domain knowledge , expressed in the form of constraints, in DL learning models to improve their performance, in particular targeting deep neural networks. We identify five (non-mutually exclusive) categories that encompass the main approaches to inject domain knowledge: 1) acting on the features space, 2) modifications to the hypothesis space, 3) data augmentation, 4) regularization schemes, 5) constrained learning.

show abstract

A Survey of Reinforcement Learning Informed by Natural Language

Cited by 24 publications

References 0 publications

Tell me why! Explanations support learning relational and causal structure

Tell me why! Explanations support learning relational and causal structure

Reinforcement Learning with Information-Theoretic Actuation

Improving Deep Learning Models via Constraint-Based Domain Knowledge: a Brief Survey

Contact Info

Product

Resources

About