Towards biologically plausible phosphene simulation for the differentiable optimization of visual cortical prostheses

Abstract

Blindness affects millions of people around the world. A promising solution to restoring a form of vision for some individuals are cortical visual prostheses, which bypass part of the impaired visual pathway by converting camera input to electrical stimulation of the visual system. The artificially induced visual percept (a pattern of localized light flashes, or ‘phosphenes’) has limited resolution, and a great portion of the field’s research is devoted to optimizing the efficacy, efficiency, and practical usefulness of the encoding of visual information. A commonly exploited method is non-invasive functional evaluation in sighted subjects or with computational models by using simulated prosthetic vision (SPV) pipelines. An important challenge in this approach is to balance enhanced perceptual realism, biologically plausibility, and real-time performance in the simulation of cortical prosthetic vision. We present a biologically plausible, PyTorch-based phosphene simulator that can run in real-time and uses differentiable operations to allow for gradient-based computational optimization of phosphene encoding models. The simulator integrates a wide range of clinical results with neurophysiological evidence in humans and non-human primates. The pipeline includes a model of the retinotopic organization and cortical magnification of the visual cortex. Moreover, the quantitative effects of stimulation parameters and temporal dynamics on phosphene characteristics are incorporated. Our results demonstrate the simulator’s suitability for both computational applications such as end-to-end deep learning-based prosthetic vision optimization as well as behavioral experiments. The modular and open-source software provides a flexible simulation framework for computational, clinical, and behavioral neuroscientists working on visual neuroprosthetics.

Gaze-contingent processing improves mobility, scene recognition and visual search in simulated head-steered prosthetic vision

“…For the real time integration of sensor data (subject pose, and gaze direction), we used the SDK (SRanipal) provided by the manufacturer. The simulation of cortical prosthetic vision was based on [30], adapted for Unity using the Cg shader programming language for graphics processing. To restrict the experimental complexity, we did not include temporal dynamics in the phosphene simulation, and we assumed full control over the phosphene brightness (with a high dynamic range).…”

Optimal Placement of High-Channel Visual Prostheses in Human Retinotopic Visual Cortex

“…For example, a dense foveal coverage would be useful for reading, or recognizing an object in front of you, while peripheral vision may be important for context awareness during navigation. In order to realistically study the functional properties of phosphene vision, a biologically and technologically plausible simulation of stimulation-evoked phosphenes is crucial (Van Der Grinten, de Ruyter van Steveninck, Lozano et al, 2024).…”