Towards a Foundation Model of the Mouse Visual Cortex

Abstract

Understanding the brain's perception algorithm is a highly intricate problem, as the inherent complexity of sensory inputs and the brain's nonlinear processing make characterizing sensory representations difficult. Recent studies have shown that functional models capable of predicting large-scale neuronal activity in response to arbitrary sensory input can be powerful tools for characterizing neuronal representations by enabling unlimited in silico experiments. However, accurately modeling responses to dynamic and ecologically relevant inputs like videos remains challenging, particularly when generalizing to new stimulus domains outside the training distribution. Inspired by recent breakthroughs in artificial intelligence, where foundation models-trained on vast quantities of data-have demonstrated remarkable capabilities and generalization, we developed a "foundation model" of the mouse visual cortex: a deep neural network trained on large amounts of neuronal responses to ecological videos from multiple visual cortical areas and mice. The model accurately predicted neuronal responses not only to natural videos but also to various new stimulus domains, such as coherent moving dots and noise patterns, as verified in vivo, underscoring its generalization abilities. The foundation model could also be adapted to new mice with minimal natural movie training data. We applied the foundation model to the MICrONS dataset: a study of the brain that integrates structure with function at unprecedented scale, containing nanometer-scale morphology, connectivity with >500,000,000 synapses, and function of >70,000 neurons within a ~1mm3 volume spanning multiple areas of the mouse visual cortex. This accurate functional model of the MICrONS data opens the possibility for a systematic characterization of the relationship between circuit structure and function. By precisely capturing the response properties of the visual cortex and generalizing to new stimulus domains and mice, foundation models can pave the way for a deeper understanding of visual computation.

An unsupervised map of excitatory neurons’ dendritic morphology in the mouse visual cortex

“…To address this question, we made use of the fact that for many of the neurons in our dataset, we have measurements of how they respond to natural stimuli [6]. We leveraged a functional digital twin – a model that accurately predicted the response of a neuron to arbitrary visual stimuli [38] – to extract a functional bar code – a vector embedding f i that describes the input-output function of a neuron analogous to how our morphological bar codes describe their morphology (Fig. 7C).…”

“…Functional digital twin can predict the functional response of the neurons to input stimuli such as natural movies. The input-output function of each neurons is described by a functional bar code f i [38]. Schematic adapted from Consortium et al [6] (Fig.…”

“…The MIC r ONS dataset encompasses EM images as well as Calcium imaging of the same portion of the visual cortex of one mouse [6]. The companion paper by Wang et al [38] created a digital twin of the functional properties of the neurons from the Calcium imaging data (Fig. 7C).…”

Bipartite invariance in mouse primary visual cortex

“…Specifically, we presented to the same neuronal population a static set of natural images as well as the identical set of natural movies used in the MICrONs data set. We then trained two different static image models: one directly on in vivo static responses, and another on in silico predicted responses to the identical static images from a recurrent neural network trained on movie stimuli (22). We found that the MEIs and DEIs generated from these two models were perceptually similar (Fig.…”

“…Second, we wanted to determine if DEIs are robust across predictive models with distinct architecture trained on different stimulus domains. To do this, we developed a method to synthesize DEIs from the dynamic digital twin trained on natural movies in the MICrONS functional connectomics data set (9,22). Specifically, we presented to the same neuronal population a static set of natural images as well as the identical set of natural movies used in the MICrONs data set.…”