Behavioral Cloning from Observation

Torabi, Faraz; Warnell, Garrett; Stone, Peter

doi:10.48550/arxiv.1805.01954

Cited by 61 publications

(85 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…IfO [40] Learning policy from aligned observation only BCO [69] Adopting IfO setting and integrating with BC TCN [61] Multi-viewpoint self-supervised IfO method One-shot IfO [5] Extracting features from unlabeled and unaligned gameplay footage Zero-Shot Visual Imitation [51] Using distance between observations to predict and penalize the actions IfO survey [72] Detailed classified recent IfO methods Imitating Latent Policies from Observation [19] Infering latent policies directly from state observations GAIfO [70] Generative adversarial structure aggregating with IfO IfO Leveraging Proprioception [71] Leveraging internal information of the agent OPOLO [81] Using dual-form of the expectation function and adversarial structure to achieve off-policy IfO follows the nature of how human and animal imitate. For example, people learn to dance by following a video, this kind of following process is achieved though detecting the changes of poses and taking actions to match the pose, which is similar to how IfO solves the problem.…”

Section: Publication Descriptionmentioning

confidence: 99%

Imitation Learning: Progress, Taxonomies and Opportunities

Zheng¹,

Verma²,

Zhou³

et al. 2021

Preprint

View full text Add to dashboard Cite

Imitation learning aims to extract knowledge from human experts' demonstrations or artificially created agents in order to replicate their behaviours. Its success has been demonstrated in areas such as video games, autonomous driving, robotic simulations and object manipulation. However, this replicating process could be problematic, such as the performance is highly dependent on the demonstration quality, and most trained agents are limited to perform well in task-specific environments. In this survey, we provide a systematic review on imitation learning. We first introduce the background knowledge from development history and preliminaries, followed by presenting different taxonomies within Imitation Learning and key milestones of the field. We then detail challenges in learning strategies and present research opportunities with learning policy from suboptimal demonstration, voice instructions and other associated optimization schemes.

show abstract

Section: Publication Descriptionmentioning

confidence: 99%

Imitation Learning: Progress, Taxonomies and Opportunities

Zheng¹,

Verma²,

Zhou³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…However, bandit algorithms assume that taking actions does not affect the state [17], while actually recommendations do have an effect on user behavior [26]; hence RL is a more suitable choice for the RS task. Another related field is imitation learning, where the policy is learned from expert demonstrations [8,9,23,34].…”

Section: Related Workmentioning

confidence: 99%

Supervised Advantage Actor-Critic for Recommender Systems

Xin¹,

Karatzoglou²,

Arapakis³

et al. 2021

Preprint

View full text Add to dashboard Cite

Casting session-based or sequential recommendation as reinforcement learning (RL) through reward signals is a promising research direction towards recommender systems (RS) that maximize cumulative profits. However, the direct use of RL algorithms in the RS setting is impractical due to challenges like off-policy training, huge action spaces and lack of sufficient reward signals. Recent RL approaches for RS attempt to tackle these challenges by combining RL and (self-)supervised sequential learning, but still suffer from certain limitations. For example, the estimation of Q-values tends to be biased toward positive values due to the lack of negative reward signals. Moreover, the Q-values also depend heavily on the specific timestamp of a sequence.To address the above problems, we propose negative sampling strategy for training the RL component and combine it with supervised sequential learning. We call this method Supervised Negative Q-learning (SNQN). Based on sampled (negative) actions (items), we can calculate the "advantage" of a positive action over the average case, which can be further utilized as a normalized weight for learning the supervised sequential part. This leads to another learning framework: Supervised Advantage Actor-Critic (SA2C). We instantiate SNQN and SA2C with four state-of-the-art sequential recommendation models and conduct experiments on two real-world datasets. Experimental results show that the proposed approaches achieve significantly better performance than state-of-the-art supervised methods and existing self-supervised RL methods . Code will be open-sourced.

show abstract

“…This formulation enables RL agents to best exploit large datasets to learn an agent that outperform the behavioral policy, since offline training is significantly cheaper compared to interacting with the environment in real-world settings. Despite the promise of the framework, current offline RL algorithms continue to under-achieve naive baselines such as Behavorial Cloning (Torabi et al, 2018), as reported by Fu et al (2020).…”

Section: Introductionmentioning

confidence: 99%

S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning

Sinha¹,

Mandlekar²,

Garg³

2021

Preprint

View full text Add to dashboard Cite

Offline reinforcement learning proposes to learn policies from large collected datasets without interaction. These algorithms have made it possible to learn useful skills from data that can then be transferred to the environment, making it feasible to deploy the trained policies in real-world settings where interactions may be costly or dangerous, such as self-driving. However, current algorithms overfit to the dataset they are trained on and perform poor out-of-distribution (OOD) generalization to the environment when deployed. We propose a Surprisingly Simple Self-Supervision algorithm (S4RL), which utilizes data augmentations from states to learn value functions that are better at generalizing and extrapolating when deployed in the environment. We investigate different data augmentation techniques that help learning a value function that can extrapolate to OOD data, and how to combine data augmentations and offline RL algorithms to learn a policy. We experimentally show that using S4RL significantly improves the state-of-the-art on most benchmark offline reinforcement learning tasks on popular benchmark datasets from D4RL (Fu et al., 2020), despite being simple and easy to implement.

show abstract

Behavioral Cloning from Observation

Cited by 61 publications

References 0 publications

Imitation Learning: Progress, Taxonomies and Opportunities

Imitation Learning: Progress, Taxonomies and Opportunities

Supervised Advantage Actor-Critic for Recommender Systems

S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning

Contact Info

Product

Resources

About