Learning Mobile Manipulation through Deep Reinforcement Learning

Abstract

Mobile manipulation has a broad range of applications in robotics. However, it is usually more challenging than fixed-base manipulation due to the complex coordination of a mobile base and a manipulator. Although recent works have demonstrated that deep reinforcement learning is a powerful technique for fixed-base manipulation tasks, most of them are not applicable to mobile manipulation. This paper investigates how to leverage deep reinforcement learning to tackle whole-body mobile manipulation tasks in unstructured environments using only on-board sensors. A novel mobile manipulation system which integrates the state-of-the-art deep reinforcement learning algorithms with visual perception is proposed. It has an efficient framework decoupling visual perception from the deep reinforcement learning control, which enables its generalization from simulation training to real-world testing. Extensive simulation and experiment results show that the proposed mobile manipulation system is able to grasp different types of objects autonomously in various simulation and real-world scenarios, verifying the effectiveness of the proposed mobile manipulation system.

Path Driven Dual Arm Mobile Co-Manipulation Architecture for Large Part Manipulation in Industrial Environments

“…Zhou et al [ 19 ] propose a mobile manipulation method integrating deep-learning-based multiple-object detection to track and grasp dynamic objects efficiently. Following this same path, Wang et al [ 20 ] present a novel mobile manipulation system that decouples visual perception from the deep reinforcement learning control, improving its generalization from simulation training to real-world testing. Additionally, Iriondo et al [ 21 ] include a deep reinforcement learning (DRL) approach for pick and place operations in logistics using a mobile manipulator.…”

A review of the challenges in mobile manipulation: systems design and RoboCup challenges

“…Manipulation in a mobile setting (see Fig. 6) is even more complex and different methods have to be developed to address flexibility [88].…”

Mobile Manipulation Integrating Enhanced AMCL High-Precision Location and Dynamic Tracking Grasp

“…With the development of deep learning, it is constantly applied to mobile operations. In reference [ 31 ], Wang et al proposed a novel mobile manipulation system, which combines the deep reinforcement learning algorithm with visual perception. Although using the visual perception advantage of deep reinforcement learning, the robot can operate the grasping task in an unstructured environment.…”