Learning the sound inventory of a complex vocal skill via an intrinsic reward

Abstract

Reward-prediction errors as reflected by dopaminergic neuron responses are thought to underlie the acquisition of multi-action skills such as birdsong or speech, where sounding similar to one's learning target is intrinsically rewarding. It is not known how the predicted reward itself is computed. Based on past behavioral studies showing context-independence and greediness of birdsong syllable inventory learning, we hypothesize a similarity-based intrinsic reward formed by a hierarchical competition between partial rewards attributed to individual syllables and calls. Assuming that developmental birdsong learning is driven by minimization of reward prediction error, we infer a zero partial reward for vocalizations that are at a large acoustic distance from a target. We train young zebra finches to learn a syllable inventory and find that this reinforcement learning model correctly predicts the exclusion of a syllable from a song and its replacement by a call. We present implications for dopaminergic responses in the avian song system during learning.