Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain

Hao, Jianye; Yang, Tianpei; Tang, Hongyao; Bai, Chenjia; Liu, Jinyi; Meng, Zhaopeng; Liu, Peng; Wang, Zhen

doi:10.1109/tnnls.2023.3236361

Cited by 41 publications

(20 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The observations can be broken down into five parts: ball information, left team data, right team data, details of the controlled player, and the match state. We employ settings similar to those used in [9], where agents can choose from 19 discrete actions, including running, sliding, shooting, and passing. We evaluate five different random seeds and average the results for presentation.…”

Section: The Performance In Grf Tasksmentioning

confidence: 99%

IPERS: Individual Prioritized Experience Replay with Subgoals for Sparse Reward Multi-Agent Reinforcement Learning

Xie,

Zhang,

Qiao

et al. 2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

Multi-agent reinforcement learning commonly uses a global team reward signal to represent overall collaborative performance. Value decomposition breaks this global reward into estimated individual value functions per agent, enabling efficient training. However, in sparse reward environments, agents struggle to assess if their actions achieve the team goal, slowing convergence. This impedes the algorithm’s convergence rate and overall efficacy. We present IPERS, an Individual Prioritized Experience Replay algorithm with Subgoals for Sparse Reward Multi-Agent Reinforcement Learning. IPERS integrates joint action decomposition and prioritized experience replay, maintaining invariance between global and individual loss gradients. Subgoals serve as intermediate goals that break down complex tasks into simpler steps with dense feedback and provide helpful intrinsic rewards that guide agents. This facilitates learning coordinated policies in challenging collaborative environments with sparse rewards. Experimental evaluations of IPERS in both the SMAC and GRF environments demonstrate rapid adaptation to diverse multi-agent tasks and significant improvements in win rate and convergence performance relative to state-of-the-art algorithms.

show abstract

Section: The Performance In Grf Tasksmentioning

confidence: 99%

IPERS: Individual Prioritized Experience Replay with Subgoals for Sparse Reward Multi-Agent Reinforcement Learning

Xie,

Zhang,

Qiao

et al. 2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

show abstract

“…Generally, rewards from the environment are extrinsic rewards, while those awarded by the exploration mechanism are intrinsic rewards. The various approaches to exploration algorithm design can be broadly categorized into four types [24]. The first is count-based exploration, where states with a higher visit count are less novel.…”

Section: Introductionmentioning

confidence: 99%

Improve exploration in deep reinforcement learning for UAV path planning using state and action entropy

Lv,

Chen,

et al. 2024

Meas. Sci. Technol.

View full text Add to dashboard Cite

Despite being a widely adopted development framework for unmanned aerial vehicle (UAV), deep reinforcement learning (DRL) is often considered sample inefficient. Particularly, UAV struggles to fully explore the state and action space in environments with sparse rewards. While some exploration algorithms have been proposed to overcome the challenge of sparse rewards, they are not specifically tailored for UAV platform. Consequently, applying those algorithms to UAV path planning may lead to problems such as unstable training processes and neglect of action space comprehension, possibly causing negative impacts on the path planning results. To address the problem of sparse rewards in UAV path planning, we propose an information-theoretic exploration algorithm, Entropy Explorer (EE), specifically for UAV platform. The proposed EE generates intrinsic rewards based on state entropy and action entropy to compensate for the scarcity of extrinsic rewards. To further improve sampling efficiency, a framework integrating EE and Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithms is proposed. Finally, the TD3-EE algorithm is tested in Airsim and compared against benchmarking algorithms. The simulation outcomes manifest that TD3-EE effectively stimulates the UAV to comprehensively explore both state and action spaces, thereby attaining superior performance compared to the benchmark algorithms in the realm of path planning.

show abstract

“…Traditional approaches to robot navigation exploration usually rely on prior information about the environment to design control strategies [3]. However, this kind of approach often performs poorly when dealing with unknown environments, such as dynamic changes and complex obstacles.…”

Section: Introductionmentioning

confidence: 99%

Field Robot Self-Exploration Based on Deep Reinforcement Learning and Safety Control Mechanism

Kang,

Zhang,

et al. 2024

Preprint

View full text Add to dashboard Cite

This paper aims to address the issues that arise when using deep reinforcement learning algorithms for reactive robotic navigation in farmland environments. These challenges include complex network models, a large number of parameters, significant computational requirements, and the possibility of being vulnerable to local optimal solutions. Moreover, the complexity of the farmland environment poses a challenge to the existing indoor reactive navigation methods, which struggle to meet the requirements of tasks such as crop protection and human cooperation.To address these issues, the paper proposes the Field Robot Autonomous Navigation System (FRANS), which utilizes Deep Reinforcement Learning (DRL) to enable farmland robots to autonomously explore unknown environments. The system consists of two modules: Global Navigation (GN) and Local Navigation (LN). The GN module establishes goal points and implements a secure control mechanism to manage the robot's actions. This approach helps to address problems like locally optimal solutions. The LN module uses a lightweight DRL framework to develop a locally navigated motion strategy that guides the robot toward waypoints. FRANS does not require any prior knowledge and is capable of adjusting the navigation strategy in real time based on the current environment. The map is generated in real time as the robot moves through the environment. Experimental results demonstrate that FRANS outperforms similar approaches.

show abstract

Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain

Cited by 41 publications

References 0 publications

IPERS: Individual Prioritized Experience Replay with Subgoals for Sparse Reward Multi-Agent Reinforcement Learning

IPERS: Individual Prioritized Experience Replay with Subgoals for Sparse Reward Multi-Agent Reinforcement Learning

Improve exploration in deep reinforcement learning for UAV path planning using state and action entropy

Field Robot Self-Exploration Based on Deep Reinforcement Learning and Safety Control Mechanism

Contact Info

Product

Resources

About