Learning user-defined sub-goals using memory editing in reinforcement learning

Abstract

The aim of reinforcement learning (RL) is to allow the agent to achieve the final goal. Most RL studies have focused on improving the efficiency of learning to achieve the final goal faster. However, the RL model is very difficult to modify an intermediate route in the process of reaching the final goal. That is, the agent cannot be under control to achieve other sub-goals in the existing studies. If the agent can go through the sub-goals on the way to the destination, the RL can be applied and studied in various fields. In this study, I propose a methodology to achieve the user-defined sub-goals as well as the final goal using memory editing. The memory editing is performed to generate various sub-goals and give an additional reward to the agent. In addition, the sub-goals are separately learned from the final goal. I set two simple environments and various scenarios in the test environments. As a result, the agent almost successfully passed the sub-goals as well as the final goal under control. Moreover, the agent was able to be induced to visit the novel state indirectly in the environments. I expect that this methodology can be used in the fields that need to control the agent in a variety of scenarios.

A Fully Controllable Agent in the Path Planning using Goal-Conditioned Reinforcement Learning

“…However, the agent of the sub-goals dedicated network did not show the shortest path whereas the agent of the policy network showed the almost shortest path. Furthermore, like a previous study [21], sometimes, the agent that learned the sub-goals was confused when the bonus point was near the current location of the agent. The reason is that the agent was trained to go through the bonus point, at first, to clear each stage.…”

“…In addition, learning user-defined sub-goals has been proposed [21]. However, the agent was partially under control.…”

“…To make the agent reach a number of goals in a dynamic environment, learning a controllable agent is needed. One of the recent approaches for controlling the agent has a limitation in that the agent can only learn the behavior from trajectories that have been directly experienced [21]. Therefore, the agent can only be under control in the area visited by the agent.…”

“…We can also get precise information by editing untidy memories [31]. A recent study used the concept of the memory editing on goal-conditioned RL for the agent to reach sub-goals so that the user can control the agent [21]. However, the agent cannot move the sub-goals in difficult environments.…”

A Controllable Agent by Subgoals in Path Planning Using Goal-Conditioned Reinforcement Learning

“…By editing the collected trajectories, we can obtain the number of trajectories. A recent study used the concept of memory editing on goal-conditioned RL for the agent to reach subgoals so that the user can control the agent by the subgoals [32]. However, the agent could not move the subgoals in difficult environments such as Fig.…”