Time-Contrastive Networks: Self-Supervised Learning from Multi-view Observation

Sermanet, Pierre; Lynch, Corey; Hsu, Jasmine; Levine, Sergey

doi:10.1109/cvprw.2017.69

Cited by 103 publications

(91 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…as real-world tasks such as cooking or assembly are inherently long-horizon and hierarchical. Recent works have attempted learning from pixel space [11,27,35,42], but learning long-horizon tasks from video in a one-shot setting remains a challenge, since both the visual learning and task complexity exacerbate the demand for better data efficiency. Our solution explicitly models the compositionality in the task structure and policy, enabling us to scale oneshot visual imitation to complex tasks.…”

Section: Conjugate Task Graphmentioning

confidence: 99%

Neural Task Graphs: Generalizing to Unseen Tasks From a Single Video Demonstration

Huang

Nair

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

105

100

View full text Add to dashboard Cite

Our goal is to generate a policy to complete an unseen task given just a single video demonstration of the task in a given domain. We hypothesize that to successfully generalize to unseen complex tasks from a single video demonstration, it is necessary to explicitly incorporate the compositional structure of the tasks into the model. To this end, we propose Neural Task Graph (NTG) Networks, which use conjugate task graph as the intermediate representation to modularize both the video demonstration and the derived policy. We empirically show NTG achieves inter-task generalization on two complex tasks: Block Stacking in Bul-letPhysics and Object Collection in AI2-THOR. NTG improves data efficiency with visual input as well as achieve strong generalization without the need for dense hierarchical supervision. We further show that similar performance trends hold when applied to real-world data. We show that NTG can effectively predict task structure on the JIGSAWS surgical dataset and generalize to unseen tasks.

show abstract

Section: Conjugate Task Graphmentioning

confidence: 99%

Neural Task Graphs: Generalizing to Unseen Tasks From a Single Video Demonstration

Huang

Nair

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

105

100

View full text Add to dashboard Cite

show abstract

“…We specifically approach this problem from the perspective of representation learning, using the learned embedding as a goal metric for reinforcement learning for reaching goal images. Prior works have aimed to learn representations for control through auto-encoding [27,48,15,16,34], pretrained supervised features [40], spatial structure [15,16,23], and viewpoint invariance [41]. However, unlike these works, we build a metric that specifically takes into account how actions lead to particular states, leading to control-centric representations that capture aspects of the observation that can be controlled, while discarding other elements.…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Visuomotor Control through Distributional Planning Networks

Yu¹,

Shevchuk²,

Sadigh³

et al. 2019

Robotics: Science and Systems XV

View full text Add to dashboard Cite

While reinforcement learning (RL) has the potential to enable robots to autonomously acquire a wide range of skills, in practice, RL usually requires manual, per-task engineering of reward functions, especially in real world settings where aspects of the environment needed to compute progress are not directly accessible. To enable robots to autonomously learn skills, we instead consider the problem of reinforcement learning without access to rewards. We aim to learn an unsupervised embedding space under which the robot can measure progress towards a goal for itself. Our approach explicitly optimizes for a metric space under which action sequences that reach a particular state are optimal when the goal is the final state reached. This enables learning effective and control-centric representations that lead to more autonomous reinforcement learning algorithms. Our experiments on three simulated environments and two real-world manipulation problems show that our method can learn effective goal metrics from unlabeled interaction, and use the learned goal metrics for autonomous reinforcement learning.

show abstract

“…An advantage of our modular design is that new advances relevant to the various stages of the pipeline can be readily incorporated to improve the overall effectiveness of the framework. However, an exciting direction for future work is to investigate methods for more end-to-end learning from visual demonstrations, for example taking inspiration from Sermanet et al [2017] and Yu et al [2018a], which may reduce the dependence on accurate pose estimators. Another exciting direction is to capitalize on our method's ability to learn from video clips and focus on large, outdoor activities, as well as motions of nonhuman animals that are conventionally very difficult, if not impossible, to mocap.…”

Section: Motion Completionmentioning

confidence: 99%

SFV

et al. 2018

Self Cite

View full text Add to dashboard Cite

Fig. 1. Simulated characters performing highly dynamic skills learned by imitating video clips of human demonstrations. Left: Humanoid performing cartwheel B on irregular terrain. Right: Backflip A retargeted to a simulated Atlas robot.Data-driven character animation based on motion capture can produce highly naturalistic behaviors and, when combined with physics simulation, can provide for natural procedural responses to physical perturbations, environmental changes, and morphological discrepancies. Motion capture remains the most popular source of motion data, but collecting mocap data typically requires heavily instrumented environments and actors. In this paper, we propose a method that enables physically simulated characters to learn skills from videos (SFV). Our approach, based on deep pose estimation and deep reinforcement learning, allows data-driven animation to leverage the abundance of publicly available video clips from the web, such as those from YouTube. This has the potential to enable fast and easy design of character controllers simply by querying for video recordings of the desired behavior. The resulting controllers are robust to perturbations, can be adapted to new settings, can perform basic object interactions, and can be retargeted to new morphologies via reinforcement learning. We further demonstrate that our method can predict potential human motions from still images, by forward simulation of learned controllers initialized from the observed pose. Our framework is able to learn a broad range of dynamic skills, including locomotion, acrobatics, and martial arts. (Video 1 )

show abstract

Time-Contrastive Networks: Self-Supervised Learning from Multi-view Observation

Cited by 103 publications

References 0 publications

Neural Task Graphs: Generalizing to Unseen Tasks From a Single Video Demonstration

Neural Task Graphs: Generalizing to Unseen Tasks From a Single Video Demonstration

Unsupervised Visuomotor Control through Distributional Planning Networks

SFV

Contact Info

Product

Resources

About