Learning Visual Servoing with Deep Features and Fitted Q-Iteration

Abstract

Visual servoing involves choosing actions that move a robot in response to observations from a camera, in order to reach a goal configuration in the world. Standard visual servoing approaches typically rely on manually designed features and analytical dynamics models, which limits their generalization capability and often requires extensive application-specific feature and model engineering.In this work, we study how learned visual features, learned predictive dynamics models, and reinforcement learning can be combined to learn visual servoing mechanisms. We focus on target following, with the goal of designing algorithms that can learn a visual servo using low amounts of data of the target in question, to enable quick adaptation to new targets. Our approach is based on servoing the camera in the space of learned visual features, rather than image pixels or manually-designed keypoints. We demonstrate that standard deep features, in our case taken from a model trained for object classification, can be used together with a bilinear predictive model to learn an effective visual servo that is robust to visual variation, changes in viewing angle and appearance, and occlusions. A key component of our approach is to use a sample-efficient fitted Q-iteration algorithm to learn which features are best suited for the task at hand. We show that we can learn an effective visual servo on a complex synthetic car following benchmark using just 20 training trajectory samples for reinforcement learning. We demonstrate substantial improvement over a conventional approach based on image pixels or hand-designed keypoints, and we show an improvement in sample-efficiency of more than two orders of magnitude over standard model-free deep reinforcement learning algorithms. Videos are available at http://rll.berkeley.edu/visual_servoing.

Neural Topological SLAM for Visual Navigation

“…The first category of tasks is ones where the goal location is known, and limited exploration is necessary. This could be in the form of a simply wandering around without colliding [15,34], following an object [23], getting to a goal coordinate [1,17]: using sequence of images along the path [5,22], or language-based instructions [2]. Sometimes, the goal is specified as an image but experience from the environment is available in the form of demonstrations [13,35], or in the form of reward-based training [25,48], which again limits the role of exploration.…”

“…Depending on the problem being considered, different representations have been investigated. For short-range locomotion tasks, purely reactive policies [3,15,23,34] suffice. For more complex problems such as target-driven navigation in a novel environment, such purely reactive strategies do not work well [48], and memory-based policies have been investigated.…”

Robotic Arm Control and Task Training through Deep Reinforcement Learning

“…[10], instead, manages to train from single view image streams a neural network able to predict the probability of successful grasps, learning thus a hand-eye coordination for grasping. Interesting related works on visual DRL for robotics are also [11], [12], [13], [14]. Data efficient DRL for DPG-based dexterous manipulation has been further explored in [15], which mainly focuses on stacking Lego blocks.…”

SE3-Pose-Nets: Structured Deep Dynamics Models for Visuomotor Control

“…Visual servoing: Finally, there have been multiple approaches to visual servoing over the years [1], [17], [18], including some newer methods that use deep learned features and reinforcement learning [19]. While these methods depend on an external system for data association or on pre-specified features, our system is trained end-to-end and can control directly from raw depth data.…”