Connecting Exploration, Generalization, and Planning in Correlated Trees

Abstract

Human reinforcement learning (RL) is characterized by different challenges. Exploration has been studied extensively in multi-armed bandits, while planning has been investigated in multi-step decision tasks. More recent work has added structured rewards to study generalization. However, past studies have often focused on a single one of these aspects, making it hard to compare results. We propose a generative model for constructing correlated trees to provide a unified and scalable method for studying exploration, planning, and generalization in a single task. In an online experiment, we find that people use structure (when provided) to generalize and perform uncertainty-directed exploration, with structure helping more in larger environments. In environments without structure, exploration becomes more random and more planning is needed. All behavioral effects are captured in a single model with recoverable parameters. In conclusion, our results connect past research on human RL in one framework using correlated trees.