Reset-free Trial-and-Error Learning for Robot Damage Recovery

Abstract

The high probability of hardware failures prevents many advanced robots (e.g., legged robots) from being confidently deployed in real-world situations (e.g., post-disaster rescue). Instead of attempting to diagnose the failures, robots could adapt by trial-and-error in order to be able to complete their tasks. In this situation, damage recovery can be seen as a Reinforcement Learning (RL) problem. However, the best RL algorithms for robotics require the robot and the environment to be reset to an initial state after each episode, that is, the robot is not learning autonomously. In addition, most of the RL methods for robotics do not scale well with complex robots (e.g., walking robots) and either cannot be used at all or take too long to converge to a solution (e.g., hours of learning). In this paper, we introduce a novel learning algorithm called "Reset-free Trial-and-Error" (RTE) that (1) breaks the complexity by pre-generating hundreds of possible behaviors with a dynamics simulator of the intact robot, and (2) allows complex robots to quickly recover from damage while completing their tasks and taking the environment into account. We evaluate our algorithm on a simulated wheeled robot, a simulated six-legged robot, and a real six-legged walking robot that are damaged in several ways (e.g., a missing leg, a shortened leg, faulty motor, etc.) and whose objective is to reach a sequence of targets in an arena. Our experiments show that the robots can recover most of their locomotion abilities in an environment with obstacles, and without any human intervention.Comment: 18 pages, 16 figures, 3 tables, 6 pseudocodes/algorithms, video at https://youtu.be/IqtyHFrb3BU, code at https://github.com/resibots/chatzilygeroudis_2018_rt

Deep Reinforcement Learning for Single-Shot Diagnosis and Adaptation in Damaged Robots

“…Algorithms based on behavior performance maps [Chatzilygeroudis et al 2018;Cully et al 2015] rely on the assumption that knowledge of the cause of damage i.e., a proper diagnosis report is not necessary to recover from the damage. Rather than considering two separate phases for damage diagnosis and recovery algorithm generation, Cully et al [Cully et al 2015], proposed a method inspired from animals, who perform trial and error to determine the least painful alternate gait in the presence of injury.…”

“…Deep Reinforcement learning (Deep RL) has been shown to be effective in modeling such navigation problems because of both its online and offline learning capabilities in high dimensional search spaces [Chatzilygeroudis et al 2018;Hwangbo et al 2017;Lobos-Tsunekawa et al 2018;Pinto et al 2017a]. In the context of adapting Authors' addresses: Shresth Verma, ABV-Indian Institute of Information Technology and Management, Gwalior, vermashresth@gmail.com; Haritha S. Nair, ABV-Indian Institute of Information Technology and Management, Gwalior, haritha1313@gmail.com; Gaurav Agarwal, ABV-Indian Institute of Information Technology and Management, Gwalior, gaurava05@gmail.com; Joydip Dhar, ABV-Indian Institute of Information Technology and Management, Gwalior, jdhar@iiitm.ac.in; Anupam Shukla, ABV-Indian Institute of Information Technology and Management, Gwalior, anupamshukla@iiitm.…”

Graph-Based Controller Synthesis for Safety-Constrained, Resilient Systems

“…In a game-theoretical setting, the FPCP can be posed as follows. (2). After this step, the game proceeds the same as Game 1.…”

“…An autonomous system operating for substantial periods of time in remote, unknown, or hostile environment will inevitably sustain damage or experience partial system failures over time due to malfunctions. Examples include unmanned aerial vehicles (UAVs) operating over contested territory [1], search-and-rescue robots [2], and rovers performing missions on extraterrestrial surfaces [3]. (b) Hostile takeover.…”

Automated Shapeshifting for Function Recovery in Damaged Robots

“…Indeed, much work has investigated how, in the absence of external supervision, a robot can automatically learn new ways to control its body when damaged [6,10,13,20,24,26,34]. While a diverse set of recovery mechanisms have been proposed, they all shared a common assumption: The damaged mechanical structure could be reconfigured, but not fundamentally deformed.…”