State representation learning for control: An overview

Lesort, Timothée; Díaz-Rodríguez, Natalia; Goudou, Jean-François; Filliat, David

doi:10.1016/j.neunet.2018.07.006

Cited by 252 publications

(174 citation statements)

References 39 publications

Supporting

Mentioning

172

Contrasting

Order By: Relevance

“…In the context of reinforcement learning (which we go into more detail in Sec. III), a good representation encodes the essential information of the state for the agent to choose its next action for a given task [40]. A compact and low-dimensional state representation can make reinforcement learning more data efficient.…”

Section: B Representation Learning For Policy Learningmentioning

confidence: 99%

See 1 more Smart Citation

Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks

et al. 2020

View full text Add to dashboard Cite

Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. It is non-trivial to manually design a robot controller that combines these modalities which have very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity. In this work, we use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our policy learning. Evaluating our method on a peg insertion task, we show that it generalizes over varying geometries, configurations, and clearances, while being robust to external perturbations. We also systematically study different self-supervised learning objectives and representation learning architectures. Results are presented in simulation and on a physical robot.

show abstract

Section: B Representation Learning For Policy Learningmentioning

confidence: 99%

“…A popular representation learning objective is reconstruction of the raw sensory input through variational autoencoders [11,29,40,70], which we consider as a baseline in this work. This unsupervised objective benefits learning stability and speed, but it is also data intensive and prone to overfitting [11].…”

Section: B Representation Learning For Policy Learningmentioning

confidence: 99%

Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Following (Lesort et al, 2018), a good state representation should be (1) Markovian (i.e., the current state summarizes all the necessary information to choose an action), (2) able to represent the robot context well enough for policy improvement, (3) able to generalize the learned value-function to unseen states with similar features, and (4), low dimensional for efficient estimation (Böhmer et al, 2015). State representation learning approaches learn low dimensional representations without direct supervision, i.e., exploiting sequences of observations, actions, rewards and generic learning objectives (Lesort et al, 2018).…”

Section: Conceptual Framework and Basic Definitionsmentioning

confidence: 99%

“…State representation learning approaches learn low dimensional representations without direct supervision, i.e., exploiting sequences of observations, actions, rewards and generic learning objectives (Lesort et al, 2018). …”

Section: Conceptual Framework and Basic Definitionsmentioning

confidence: 99%

Open-Ended Learning: A Conceptual Framework Based on Representational Redescription

et al. 2018

Self Cite

View full text Add to dashboard Cite

Reinforcement learning (RL) aims at building a policy that maximizes a task-related reward within a given domain. When the domain is known, i.e., when its states, actions and reward are defined, Markov Decision Processes (MDPs) provide a convenient theoretical framework to formalize RL. But in an open-ended learning process, an agent or robot must solve an unbounded sequence of tasks that are not known in advance and the corresponding MDPs cannot be built at design time. This defines the main challenges of open-ended learning: how can the agent learn how to behave appropriately when the adequate states, actions and rewards representations are not given? In this paper, we propose a conceptual framework to address this question. We assume an agent endowed with low-level perception and action capabilities. This agent receives an external reward when it faces a task. It must discover the state and action representations that will let it cast the tasks as MDPs in order to solve them by RL. The relevance of the action or state representation is critical for the agent to learn efficiently. Considering that the agent starts with a low level, task-agnostic state and action spaces based on its low-level perception and action capabilities, we describe open-ended learning as the challenge of building the adequate representation of states and actions, i.e., of redescribing available representations. We suggest an iterative approach to this problem based on several successive Representational Redescription processes, and highlight the corresponding challenges in which intrinsic motivations play a key role.

show abstract

“…These methods typically learn a nonlinear embedding (e.g., via autoencoders [29,32,26,27]), and-inspired by Koopman operator theory-learn a dynamics model that is constrained to be linear. In a control [24] or reinforcementlearning context [4], the embedding and dynamics models can be learned simultaneously from observations of the state, but most approaches restrict the dynamics to be locally linear [14,19,33,2]. Preprint. The second class of methods corresponds to projection-based dynamics learning (often referred to as "model reduction"), which learns the embedding in a data-driven manner, but computes the dynamics model via a projection process executed on the governing system of ODEs (which must be explicitly known).…”

Section: Introductionmentioning

confidence: 99%

Deep Conservation: A latent dynamics model for exact satisfaction of physical conservation laws [Report]

Lee

Carlberg

2019

View full text Add to dashboard Cite

This work proposes an approach for latent dynamics learning that exactly enforces physical conservation laws. The method comprises two steps. First, we compute a low-dimensional embedding of the high-dimensional dynamical-system state using deep convolutional autoencoders. This defines a low-dimensional nonlinear manifold on which the state is subsequently enforced to evolve. Second, we define a latent dynamics model that associates with a constrained optimization problem. Specifically, the objective function is defined as the sum of squares of conservation-law violations over control volumes in a finite-volume discretization of the problem; nonlinear equality constraints explicitly enforce conservation over prescribed subdomains of the problem. The resulting dynamics model-which can be considered as a projection-based reduced-order model-ensures that the time-evolution of the latent state exactly satisfies conservation laws over the prescribed subdomains. In contrast to existing methods for latent dynamics learning, this is the only method that both employs a nonlinear embedding and computes dynamics for the latent state that guarantee the satisfaction of prescribed physical properties. Numerical experiments on a benchmark advection problem illustrate the method's ability to significantly reduce the dimensionality while enforcing physical conservation. arXiv:1909.09754v1 [physics.comp-ph]

show abstract

State representation learning for control: An overview

Cited by 252 publications

References 39 publications

Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks

Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks

Open-Ended Learning: A Conceptual Framework Based on Representational Redescription

Deep Conservation: A latent dynamics model for exact satisfaction of physical conservation laws [Report]

Contact Info

Product

Resources

About