Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration

Abstract

Curiosity-based reward schemes can present powerful exploration mechanisms which facilitate the discovery of solutions for complex, sparse or long-horizon tasks. However, as the agent learns to reach previously unexplored spaces and the objective adapts to reward new areas, many behaviours emerge only to disappear due to being overwritten by the constantly shifting objective. We argue that merely using curiosity for fast environment exploration or as a bonus reward for a specific task does not harness the full potential of this technique and misses useful skills. Instead, we propose to shift the focus towards retaining the behaviours which emerge during curiosity-based learning. We posit that these self-discovered behaviours serve as valuable skills in an agent's repertoire to solve related tasks. Our experiments demonstrate the continuous shift in behaviour throughout training and the benefits of a simple policy snapshot method to reuse discovered behaviour for transfer tasks.

Towards Agent-Based Testing of 3D Games using Reinforcement Learning

“…One of the first attempts to consider reinforcement learning agents with a sense of curiosity is [7,19], where curiosity is formulated as the error in an agent's ability to predict the consequence of its actions. The curiosity-powered agent learns how to interact with the environment by curiosity alone and able to learn skills to finish the game-play [11,12,17,20,25,28]. Authors of [10] address the problem of automatically exploring and testing 3D games using RL.…”

Physical Derivatives: Computing policy gradients by physical forward-propagation

“…Despite traditional RL that the learning is driven by an extrinsic reward signal, intrinsically motivated RL concerns task-agnostic learning (Sontakke et al, 2021b,a). Similar to animals' babies (Touwen et al, 1992), the agent may undergo a developmental period in which it acquires reusable modular skills (Kaplan and Oudeyer, 2003;Weng et al, 2001;Tian et al, 2021), such as curiosity and confidence (Schmidhuber, 1991a;Kompella et al, 2017;Burda et al, 2018;Mirza et al, 2020;Groth et al, 2021;Huang et al, 2022). Another aspect of such general competence is the ability of the agent to remain safe during its learning and deployment period (Garcıa and Fernández, 2015).…”

Neuron-level Prediction and Noise can Implement Flexible Reward-Seeking Behavior

“…However, predictive agents are susceptible to the "dark room problem," where agents minimize predictive errors by either reducing their activity to zero or staying in places where nothing happens [42]. Predictive agents that explore and act in environments usually need additional components to work, such as separate action selection modules [13,38] or curiosity drives [18]. [36].…”