Modularity benefits reinforcement learning agents with competing homeostatic drives

Abstract

The problem of balancing conflicting needs is fundamental to intelligence. Standard reinforcement learning algorithms maximize a scalar reward, which requires combining different objective-specific rewards into a single number. Alternatively, different objectives could also be combined at the level of action value, such that specialist modules responsible for different objectives submit different action suggestions to a decision process, each based on rewards that are independent of one another. In this work, we explore the potential benefits of this alternative strategy. We investigate a biologically relevant multi-objective problem, the continual homeostasis of a set of variables, and compare a monolithic deep Q-network to a modular network with a dedicated Q-learner for each variable. We find that the modular agent: a) requires minimal exogenously determined exploration; b) has improved sample efficiency; and c) is more robust to out-of-domain perturbation.

Reward-Bases: Dopaminergic Mechanisms for Adaptive Acquisition of Multiple Reward Types

“…There is also growing evidence that dopamine neurons are sensitive to various aspects of the state-space such as locomotion and kinematic behaviour (Barter et al, 2015;Dodson et al, 2016;Engelhard et al, 2019;Kremer, Flakowski, Rohner, & Lüscher, 2020) as well as choice behaviour (Coddington & Dudman, 2018;Parker et al, 2016) and indeed that this heterogeneity is topographically organized due to spatially organized cortical projections (Engelhard et al, 2019;Howe & Dombeck, 2016). Additionally, recent theoretic work has demonstrated that RL agents including separate modules predicting reward in different dimensions have advantages in task requiring acquisition of multiple resource types (Dulberg, Dubey, Berwian, & Cohen, 2022).…”