NeuroMechFly v2, simulating embodied sensorimotor control in adultDrosophila

Abstract

Discovering the principles underlying the neural and biomechanical control of animal behavior requires a tight dialogue between real experiments and data-driven neuromechanical models. Until now, such models have primarily been used to further our understanding of lower-level motor control. For most whole-animal simulations, we still lack an effective framework for studying how the brain processes environmental signals to regulate motor behavior. The adult fly, Drosophila melanogaster, is well-suited for data-driven modeling and can be simulated using the neuromechanical model, NeuroMechFly. However, until now this simulation framework did not permit the exploration of full hierarchical sensorimotor loops. Here we present NeuroMechFly 2.0, a framework that greatly expands whole-animal modeling of Drosophila by enabling visual and olfactory processing as well as complex three-dimensional environments that can be navigated using leg adhesion. To illustrate its capabilities we explore the effectiveness of biologically-inspired leg controllers for navigating diverse terrain, and show how one can build and use Reinforcement Learning to train an end-to-end hierarchical model with multimodal sensory processing, descending commands, and low-level motor control in closed loop. NeuroMechFly 2.0 can accelerate the discovery of explanatory models of the nervous system and the development of machine learning models to control autonomous artificial agents and robots.

Tools for connectomic reconstruction and analysis of a femaleDrosophilaventral nerve cord

“…For example, it will enable the identification and analysis of neural circuits for descending modulation (Aymanns et al, 2022; Namiki et al, 2018), ascending communication with the brain (Chen et al, 2022; Cheong et al, 2024), and sensory organs distributed across the fly limbs, thorax, and abdomen (Tuthill and Wilson, 2016). By creating a bridge between the VNC connectome and the body, the MN projection map will facilitate development and analysis of neuromechanical models for flexible motor control (Lobato-Rios et al, 2022; Wang-Chen et al, 2023). The compact sensorimotor circuits that mediate robust control of the fly leg and wing may provide inspiration for engineering of micro-scale robotic systems (Dallmann et al, 2023).…”

Sensorimotor delays constrain robust locomotion in a 3D kinematic model of fly walking

“…The formulation of a modular, multi-layered model for locomotor control makes new experimentally-testable hypotheses about fly motor control and can also be applied to investigate limbed locomotion in other organisms. Future extensions of the model include inclusion of premotor neural circuits from the fly connectome (Lesser et al, 2023; Cheong et al, 2023) and biomechanical interactions between the limb and the environment (Lobato-Rios et al, 2022; Wang-Chen et al, 2023; Vaxenburg et al, 2024).…”

“…A promising avenue for future investigation is integration of our controller architecture with a virtual physics model (Lobato-Rios et al, 2022; Wang-Chen et al, 2023; Vaxenburg et al, 2024), which would facilitate incorporation of dynamical coupling between legs, as well as leg-ground contact interactions. The inclusion of these features may require additional coordination between the legs, which might decrease allowable values of sensory and motor delay.…”

“…The second approach focuses on physical details of the legs and its joints. Schilling et al (2013) uses a decentralized reactive controller to recreate hexapod walking patterns; Goldsmith et al (2019) and Goldsmith et al (2020) introduce a robotic platform; Lobato-Rios et al (2022), Wang-Chen et al (2023), and Vaxenburg et al (2024) develop virtual fly simulations in a physics engine. Walking behavior in these models are also driven by coupled oscillators with tuned parameters that reproduce inter-leg coordination patterns; however, the underlying joint kinematics are typically unrealistic.…”

Mechanosensory and command contributions to theDrosophilagrooming sequence

“…This generates the inversion of the leg to contact the thorax and the anterior to posterior sweeping movement that characterize thoracic grooming. Future work will include biomechanical modeling (NeuroMechFly 2.0: Wang-Chen et al 2023) using information from these potential motor control circuits and functional tests of effects on limb kinematics using DeepLabCut (Mathis et al 2018).…”