Automated aerial suspended cargo delivery through reinforcement learning

Abstract

Cargo-bearing unmanned aerial vehicles (UAVs) have tremendous potential to assist humans by delivering food, medicine, and other supplies. For time-critical cargo delivery tasks, UAVs need to be able to quickly navigate their environments and deliver suspended payloads with bounded load displacement. As a constraint balancing task for joint UAV-suspended load system dynamics, this task poses a challenge. This article presents a reinforcement learning approach for aerial cargo delivery tasks in environments with static obstacles. We first learn a minimal residual oscillations task policy in obstacle-free environments using a specifically designed feature vector for value function approximation that allows generalization beyond the training domain. The method works in continuous state and discrete action spaces. Since planning for aerial cargo requires very large action space (over 10 6 actions) that is impractical for learning, we define formal conditions for a class of robotics problems where learning can occur in a simplified problem space and successfully transfer to a broader problem space. Exploiting these guarantees and relying on the discrete action space, we learn the swing-free policy in a subspace several orders of magnitude smaller, and later develop a method for swing-free trajectory planning along a path. As an extension to tasks in environments with static obstacles where the load displacement needs to be bounded throughout the trajectory, sampling-based motion planning generates collision-free paths. Next, a reinforcement learning agent transforms these paths into trajectories that maintain the bound on the load displacement while following the collision-free path in a timely manner. We verify the approach both in simulation and in experiments on a quadrotor with suspended load and verify the method's safety and feasibility through a demonstration where a quadrotor delivers an open container of liquid to a human subject. The contributions of this work are twofold. First, this article presents a solution to a challenging, and vital problem of planning a constraint-balancing task for an inherently unstable non-linear system in the presence of obstacles. Second, AI and robotics researchers can both benefit from the provided theoretical guarantees of system stability on a class of constraint-balancing tasks that occur in very large action spaces.

Avoiding moving obstacles with stochastic hybrid dynamics using PEARL: PrEference Appraisal Reinforcement Learning

“…Since manually accounting for all possibilities is often infeasible, sampling-based, learning-based, and other intelligent methods are the norm [18]. Reinforcement learning (RL), in particular, has been successful for robotic task learning [15] in several problems such as table tennis [21], swing-free UAV delivery [7], and a self-driving car [12]. However, traditional RL methods do not handle continuous and high-dimensional state spaces well [11].…”

“…Because the method learns and performs the task in the feature space, such transfer is possible. Previously, we empirically showed that, using hand-crafted features, batch RL learns in small spaces and acts on larger problems [8], [7], but did not address when learning transfer is possible and how to do it for an arbitrary problem. This paper formalizes the feature selection and the conditions under which transfer is possible so that it can be applied for classes of PBTs.…”

Towards a science of integrated AI and Robotics

“…Papers [13] and [16] also deal with learning, but put a strong emphasis on managing the complexity of doing so, thus addressing RC3. [13] introduces a scalable methodology to learn and transfer knowledge of the transition (and reward) models for model-based reinforcement learning in a complex world.…”

“…The authors use a formulation of Markov decision processes that support efficient online-learning of relevant problem features in order to approximate world dynamics. [16] presents a reinforcement learning approach for aerial cargo delivery tasks in environments with static obstacles. The authors plan and create swing-free trajectories with bounded load displacements.…”

Human Demonstrations for Fast and Safe Exploration in Reinforcement Learning

“…Reinforcement learning has already been applied to quadrotors with a slung load. 3,[12][13][14] All these studies use a model of the system dynamics. In this paper, a two-dimensional version of this slung load problem is used with a model-free reinforcement learning controller.…”