Reinforcement Learning for Improving Agent Design

Ha, David

doi:10.1162/artl_a_00301

Cited by 82 publications

(63 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, their method results in only simple robots with a few components, and assumes physically implausible reconfigurability. Reinforcement learning has also been applied in order to continually improve agent design [Ha 2018]. In their design framework, both the environment and robot are altered to enable the agent to learn more effectively.…”

Section: Generative Design For Roboticsmentioning

confidence: 99%

RoboGrammar

et al. 2020

View full text Add to dashboard Cite

We present RoboGrammar , a fully automated approach for generating optimized robot structures to traverse given terrains. In this framework, we represent each robot design as a graph, and use a graph grammar to express possible arrangements of physical robot assemblies. Each robot design can then be expressed as a sequence of grammar rules. Using only a small set of rules our grammar can describe hundreds of thousands of possible robot designs. The construction of the grammar limits the design space to designs that can be fabricated. For a given input terrain, the design space is searched to find the top performing robots and their corresponding controllers. We introduce Graph Heuristic Search - a novel method for efficient search of combinatorial design spaces. In Graph Heuristic Search, we explore the design space while simultaneously learning a function that maps incomplete designs (e.g., nodes in the combinatorial search tree) to the best performance values that can be achieved by expanding these incomplete designs. Graph Heuristic Search prioritizes exploration of the most promising branches of the design space. To test our method we optimize robots for a number of challenging and varied terrains. We demonstrate that RoboGrammar can successfully generate nontrivial robots that are optimized for a single terrain or a combination of terrains.

show abstract

Section: Generative Design For Roboticsmentioning

confidence: 99%

RoboGrammar

et al. 2020

View full text Add to dashboard Cite

show abstract

“…For example, in voxel-based robot simulators, [61,62] which use voxels as structural building blocks, there are 4.5 × 10 8 unique ways to arrange 12 voxels to form a robot, and the design space (the number of possible designs) increases exponentially with each additional block. [63] As a result, evolutionary [64][65][66][67] and learning [68,69] algorithms are usually employed to efficiently explore the vast space of possible robot designs. [70] Directly incorporating biologically inspired mechanisms of shape change-for example, slowly extruding limbs during optimization rather than optimizing controllers only for the final, legged form of the robot-has been shown to speed the evolution of robust, adaptive behavior in simulated robots.…”

Section: Simulated Shape Changing Robotsmentioning

confidence: 99%

Shape Changing Robots: Bioinspiration, Simulation, and Physical Realization

et al. 2020

View full text Add to dashboard Cite

are specialized for a single task, and cannot adapt their body to accomplish additional tasks after manufacture. Moreover, biological bodies are often highly regenerative, and able to repair and reconfigure their large-scale architecture in the face of significant damage or radical changes to their components. [3] For example, salamanders regenerate amputated limbs, [4] and fragments cut from arbitrary portions of planaria flatworms can rebuild (and rescale) their bodies to recover a full, correct anatomy. [3] Remarkably, many of these systems are able to retain information, such as learned memories, despite drastic reconfiguration or total replacement of their brains. [5] In these integrated living systems, intelligence, memory, learning, behavior, and body structure are all intertwined and emerge from the multiscale dynamics of the same robust and highly fault-tolerant medium.Evolution did not result in hard-coded body plans purely determined by genetic factors, but rather produced diverse examples of intelligent self-modifying systems which adapt to numerous extragenomic influences. [6] In this way, biology serves as an important proof-of-principle, and design challenge, for artificial intelligence and shape changing robots. Despite having access to this extensive set of model systems, the realization of general-purpose, adaptive robots has remained elusive. Researchers have proposed modular robots that can be attached to each other to expand functionality, [7] passively conforming universal grippers, [8] reconfigurable robotic skins, [9] self-assembling robot swarms, [10] gait-switching mechanisms [11] and controllers, [12,13] and algorithms that quickly re-adapt to multiple distinct tasks. [14] Such approaches succeed at adaptation but operate under the assumption that the robot's body is only reconfigured or reshaped due to external forces, and do not explore the possibility of synthetic machines that actively grow, regenerate, deform, or otherwise change the resting shape of their constituent components.With the introduction of a conformable gripper by Hirose in 1978, [15] followed by continuum robot arms, [16] silicone grippers, [17] and variable stiffness actuators, [18] robots that can adapt to real-world environments by changing their shape are becoming closer to reality. In particular, the idea of passively conforming around objects during grasping has been quite successful. [17,19,20] Soft robots have shown potential in other applications, including human-robot interaction and exploration, as reviewed by Kim et al., [21] Rus et al., [22] and others. [23,24] For a comprehensive review of the role of deformation in singlefunction soft robots, the reader is referred to Wang et al. [25] One of the key differentiators between biological and artificial systems is the dynamic plasticity of living tissues, enabling adaptation to different environmental conditions, tasks, or damage by reconfiguring physical structure and behavioral control policies. Lack of dynamic plasticity is a significant limitation for a...

show abstract

“…Deep RL has recently been used for robot design (Schaff et al 2018;Ha 2018) to simultaneously learn a design and a control policy. Deep RL requires only a sparse reward function be formulated for each task.…”

Section: Related Workmentioning

confidence: 99%

“…We compare our approach to two related methods which search for modular arrangements: a best-first search (Ha et al 2018) and an evolutionary search (Icer et al 2017). After training the DQN, our algorithm finds lower-cost solutions more efficiently than these related methods.…”

Section: Introductionmentioning

confidence: 99%

Modular Robot Design Synthesis with Deep Reinforcement Learning

Whitman

Bhirangi

Travers

et al. 2020

AAAI

View full text Add to dashboard Cite

Modular robots hold the promise of versatility in that their components can be re-arranged to adapt the robot design to a task at deployment time. Even for the simplest designs, determining the optimal design is exponentially complex due to the number of permutations of ways the modules can be connected. Further, when selecting the design for a given task, there is an additional computational burden in evaluating the capability of each robot, e.g., whether it can reach certain points in the workspace. This work uses deep reinforcement learning to create a search heuristic that allows us to efficiently search the space of modular serial manipulator designs. We show that our algorithm is more computationally efficient in determining robot designs for given tasks in comparison to the current state-of-the-art.

show abstract

Reinforcement Learning for Improving Agent Design

Cited by 82 publications

References 66 publications

RoboGrammar

RoboGrammar

Shape Changing Robots: Bioinspiration, Simulation, and Physical Realization

Modular Robot Design Synthesis with Deep Reinforcement Learning

Contact Info

Product

Resources

About