Gamer in the scanner : Event-related analysis of fMRI activity during retro videogame play guided by automated annotations of game content

Abstract

In recent years, videogames have gathered interest in cognitive neuroscience for their potential to study cognition in dynamical and naturalistic contexts. Yet, inherent game complexity can push traditional modeling to its limits, and current annotation methodologies may not fully capture the depth of their content. Neuroimaging studies have resorted to manual annotations or employed modified game versions that offer greater experimental control, both approaches limiting and labor-intensive. Here, we introduce an innovative approach using the gym-retro Python library to emulate a retro action-platformer, Shinobi III: Return of the Ninja Master (Sega, 1993), which enables automated annotations from the game's memory states. We demonstrate how this method facilitates the study of the neural basis of game events, including player actions (jumping, hitting) and feedback events (killing an enemy or being hit). Four individuals played Shinobi while undergoing functional magnetic resonance magnetic imaging (fMRI). We used automated game annotation to derive event-based general linear models of the BOLD signal in each subject, and showed common and subject-specific activation patterns in visual and sensori-motor cortices. Multi-voxel pattern analysis (MVPA) conducted on session-level annotation-derived activation maps achieved highly accurate brain decoding across all annotations. Given that the gym-retro emulation platform supports over a thousand games, this study underscores the potential of automated game annotations to study complex cognitive processes in dynamic settings across a broad game spectrum.