Interesting Object, Curious Agent: Learning Task-Agnostic Exploration

Abstract

Common approaches for task-agnostic exploration learn tabula-rasa -the agent assumes isolated environments and no prior knowledge or experience. However, in the real world, agents learn in many environments and always come with prior experiences as they explore new ones. Exploration is a lifelong process. In this paper, we propose a paradigm change in the formulation and evaluation of taskagnostic exploration. In this setup, the agent first learns to explore across many environments without any extrinsic goal in a task-agnostic manner. Later on, the agent effectively transfers the learned exploration policy to better explore new environments when solving tasks. In this context, we evaluate several baseline exploration strategies and present a simple yet effective approach to learning taskagnostic exploration policies. Our key idea is that there are two components of exploration: (1) an agent-centric component encouraging exploration of unseen parts of the environment based on an agent's belief; (2) an environment-centric component encouraging exploration of inherently interesting objects. We show that our formulation is effective and provides the most consistent exploration across several training-testing environment pairs. We also introduce benchmarks and metrics for evaluating task-agnostic exploration strategies. The source code is available at https://github.com/sparisi/cbet/.

Spatial and Temporal Hierarchy for Autonomous Navigation Using Active Inference in Minigrid Environment

“…Baseline . To establish a baseline for the navigation tasks, we compare our method against: C-BET [ 16 ], an RL algorithm combining model-based planning with uncertainty estimation for efficient exploration and decision-making. Random network distillation (RND) [ 58 ], integrates intrinsic curiosity-driven exploration to incentivise the agent’s visitation of novel states, meant to foster a deeper understanding of the environment.…”

“…C-BET [ 16 ], an RL algorithm combining model-based planning with uncertainty estimation for efficient exploration and decision-making.…”

“…Our experimental evaluation in mini-grid room maze environments showcases the effectiveness of our method in exploration and goal-related tasks. When compared to other reinforcement learning (RL) models such as C-Bet [ 16 ], Count [ 60 ], Curiosity [ 59 ], RND [ 58 ], and DreamerV3 [ 5 ], our hierarchical active inference world model consistently demonstrates competitive performance in exploration speed and coverage, as well as efficiency in reaching goals. Moreover, qualitative assessments show how accurate the cognitive map can be compared to the real environment and how the agent is able to differentiate aliased locations and use information gain to optimise navigation.…”

“…Moreover, despite the success of recent machine learning, these techniques typically require a considerable amount of training data to build accurate environment models. This training data can be obtained from simulation [ 26 , 27 ]; provided by humans (either by labelling, as in the works in [ 28 , 29 ] or by demonstration, as in [ 30 ]); or by gathering data in an experimental setting [ 16 , 31 , 32 ]. These methods all aim to predict the consequences of actions in the environment but typically generalise poorly across environments.…”

“…To overcome the limitations of these non-neural network approaches, recent advancements have focused on utilising neural networks for navigation [ 5 , 16 , 17 , 18 ]. Neural-network-based models, trained on large datasets, have shown promise in learning navigational policies directly from raw sensory input.…”