Exploration in Deep Reinforcement Learning: A Comprehensive Survey

Yang, Tianpei; Tang, Hongyao; Bai, Chenjia; Liu, Jinyi; Hao, Jianye; Meng, Zhaopeng; Liu, Peng

doi:10.48550/arxiv.2109.06668

Cited by 8 publications

(14 citation statements)

References 85 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also believe the exploration benefits we observed are novel. The problem of exploration in RL is fundamental, and existing solutions make use of noise, explicit exploratory drives/bonuses, and/or other forms of auto-annealing [9,10]. We suggest an additional class of strategies, namely, exploration as an added benefit of having multiple independent drives, since exploitation from the perspective of one module is exploration from the perspective of another.…”

Section: Discussionmentioning

confidence: 99%

Modularity benefits reinforcement learning agents with competing homeostatic drives

Dulberg¹,

Dubey²,

Berwian³

et al. 2022

Preprint

View full text Add to dashboard Cite

The problem of balancing conflicting needs is fundamental to intelligence. Standard reinforcement learning algorithms maximize a scalar reward, which requires combining different objective-specific rewards into a single number. Alternatively, different objectives could also be combined at the level of action value, such that specialist modules responsible for different objectives submit different action suggestions to a decision process, each based on rewards that are independent of one another. In this work, we explore the potential benefits of this alternative strategy. We investigate a biologically relevant multi-objective problem, the continual homeostasis of a set of variables, and compare a monolithic deep Q-network to a modular network with a dedicated Q-learner for each variable. We find that the modular agent: a) requires minimal exogenously determined exploration; b) has improved sample efficiency; and c) is more robust to out-of-domain perturbation.

show abstract

Section: Discussionmentioning

confidence: 99%

Modularity benefits reinforcement learning agents with competing homeostatic drives

Dulberg¹,

Dubey²,

Berwian³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…However, these works are outdated as more CDL methods have emerged and become increasingly popular since then. In the domain of reinforcement learning, the recent success of leveraging intrinsic rewards (artificial curiosity) to encourage efficient exploration is reviewed and discussed [4,143]. Nevertheless, these works lack a comprehensive framework and nuanced understanding of psychological curiosity to differentiate and utilize different types of artificial curiosity fundamentally.…”

Section: Introductionmentioning

confidence: 99%

From Psychological Curiosity to Artificial Curiosity: Curiosity-Driven Learning in Artificial Intelligence Tasks

Sun¹,

Qian²,

Chen³

2022

Preprint

View full text Add to dashboard Cite

Psychological curiosity plays a significant role in human intelligence to enhance learning through exploration and information acquisition. In the Artificial Intelligence (AI) community, artificial curiosity provides a natural intrinsic motivation for efficient learning as inspired by human cognitive development; meanwhile, it can bridge the existing gap between AI research and practical application scenarios, such as overfitting, poor generalization, limited training samples, high computational cost, etc. As a result, curiosity-driven learning (CDL) has become increasingly popular, where agents are self-motivated to learn novel knowledge. In this paper, we first present a comprehensive review on the psychological study of curiosity and summarize a unified framework for quantifying curiosity as well as its arousal mechanism. Based on the psychological principle, we further survey the literature of existing CDL methods in the fields of Reinforcement Learning, Recommendation, and Classification, where both advantages and disadvantages as well as future work are discussed. As a result, this work provides fruitful insights for future CDL research and yield possible directions for further improvement.

show abstract

“…The key to solve this problem is to reduce the size of the search space properly (Yang et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

API: Boosting Multi-Agent Reinforcement Learning via Agent-Permutation-Invariant Networks

Hao¹,

Wang²,

Mao³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Multi-agent reinforcement learning suffers from poor sample efficiency due to the exponential growth of the state-action space. Considering a homogeneous multiagent system, a global state consisting of m homogeneous components has m! differently ordered representations, thus designing functions satisfying permutation invariant (PI) can reduce the state space by a factor of 1 m! . However, mainstream MARL algorithms ignore this property and learn over the original state space. To achieve PI, previous works including data augmentation based methods and embeddingsharing architecture based methods, suffer from training instability and limited model capacity. In this work, we propose two novel designs to achieve PI, while avoiding the above limitations. The first design permutes the same but differently ordered inputs back to the same order and the downstream networks only need to learn function mapping over fixed-ordering inputs instead of all permutations, which is much easier to train. The second design applies a hypernetwork to generate customized embedding for each component, which has higher representational capacity than the previous embedding-sharing method. Empirical results on the SMAC benchmark show that the proposed method achieves 100% win-rates in almost all hard and super-hard scenarios (never achieved before), and superior sample-efficiency than the state-of-the-art baselines by up to 400%.

show abstract

Exploration in Deep Reinforcement Learning: A Comprehensive Survey

Cited by 8 publications

References 85 publications

Modularity benefits reinforcement learning agents with competing homeostatic drives

Modularity benefits reinforcement learning agents with competing homeostatic drives

From Psychological Curiosity to Artificial Curiosity: Curiosity-Driven Learning in Artificial Intelligence Tasks

API: Boosting Multi-Agent Reinforcement Learning via Agent-Permutation-Invariant Networks

Contact Info

Product

Resources

About