Deep Learning–Based Perceptual Stimulus Encoder for Bionic Vision

Relic, Lucas; Zhang, Bowen; Tuan, Yi-Lin; Beyeler, Michael

doi:10.1145/3519391.3524034

Cited by 13 publications

(16 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Surrogate Model: We also implemented the surrogate approach from [38] as a baseline method, where f is approximated with another deep neural network fφ (s; θ f ) with weights θ f . To achieve this we generated 50,000 percepts using randomly selected stimuli passed through f and fit a deep neural network to produce identical images.…”

Section: Methodsmentioning

confidence: 99%

“…Our implementation improves upon [38] by using the more advanced phosphene model described above, which accounts for effects of stimulus properties and allows for optimization of stimulus frequency in addition to amplitude.…”

Section: Methodsmentioning

confidence: 99%

“…Datasets We first evaluated on handwritten digits from MNIST [13], enabling comparison to previous works [38]. Images preprocessing consisted of resizing the target images to the same shape as the output of f (49x49).…”

Section: Methodsmentioning

confidence: 99%

“…Relic et al [38] also utilized the end-to-end approach, but without the secondary decoder network used in [11]. They used a more realistic phosphene model, which accounts for some spatial distortions (e.g., axonal streaks), but not the effects of stimulus parameters.…”

Section: Related Workmentioning

confidence: 99%

“…In order to optimize end-to-end, the forward model must be differentiable and computationally efficient. When this is not the case, an alternative approach is to train a surrogate neural network, f , to approximate the forward model [38,31,33,34]. However, even well-trained surrogates may result in errors when included in our end-to-end framework, due to the encoders' ability to learn to exploit the surrogate model.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

A Hybrid Neural Autoencoder for Sensory Neuroprostheses and Its Applications in Bionic Vision

Granley¹,

Relic²,

Beyeler³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Sensory neuroprostheses are emerging as a promising technology to restore lost sensory function or augment human capacities. However, sensations elicited by current devices often appear artificial and distorted. Although current models can often predict the neural or perceptual response to an electrical stimulus, an optimal stimulation strategy solves the inverse problem: what is the required stimulus to produce a desired response? Here we frame this as an end-to-end optimization problem, where a deep neural network encoder is trained to invert a known, fixed forward model that approximates the underlying biological system. As a proof of concept, we demonstrate the effectiveness of our hybrid neural autoencoder (HNA) on the use case of visual neuroprostheses. We found that HNA is able to produce high-fidelity stimuli from the MNIST and COCO datasets that outperform conventional encoding strategies and surrogate techniques across all tested conditions. Overall this is an important step towards the long-standing challenge of restoring high-quality vision to people living with incurable blindness and may prove a promising solution for a variety of neuroprosthetic technologies.Preprint. Under review.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A Hybrid Neural Autoencoder for Sensory Neuroprostheses and Its Applications in Bionic Vision

Granley¹,

Relic²,

Beyeler³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Neuromorphic hardware for somatosensory neuroprostheses

Donati,

Valle

2024

Nat Commun

View full text Add to dashboard Cite

In individuals with sensory-motor impairments, missing limb functions can be restored using neuroprosthetic devices that directly interface with the nervous system. However, restoring the natural tactile experience through electrical neural stimulation requires complex encoding strategies. Indeed, they are presently limited in effectively conveying or restoring tactile sensations by bandwidth constraints. Neuromorphic technology, which mimics the natural behavior of neurons and synapses, holds promise for replicating the encoding of natural touch, potentially informing neurostimulation design. In this perspective, we propose that incorporating neuromorphic technologies into neuroprostheses could be an effective approach for developing more natural human-machine interfaces, potentially leading to advancements in device performance, acceptability, and embeddability. We also highlight ongoing challenges and the required actions to facilitate the future integration of these advanced technologies.

show abstract

Towards a Smart Bionic Eye: AI-powered artificial vision for the treatment of incurable blindness

Beyeler¹,

Sanchez-Garcia²

2022

J. Neural Eng.

Self Cite

View full text Add to dashboard Cite

Objective. How can we return a functional form of sight to people who are living with incurable blindness? Despite recent advances in the development of visual neuroprostheses, the quality of current prosthetic vision is still rudimentary and does not differ much across different device technologies. Approach. Rather than aiming to represent the visual scene as naturally as possible, a Smart Bionic Eye could provide visual augmentations through the means of artificial intelligence (AI)-based scene understanding, tailored to specific real-world tasks that are known to affect the quality of life of people who are blind, such as face recognition, outdoor navigation, and self-care. Main results. Complementary to existing research aiming to restore natural vision with these devices, we propose a patient-centered iterative approach to provide visual augmentations designed to support a range of patient-relevant real-world tasks. Significance. The ability of a visual prosthesis to support everyday tasks might make the difference between abandoned technology and a widely adopted next-generation neuroprosthetic device.

show abstract

Deep Learning–Based Perceptual Stimulus Encoder for Bionic Vision

Cited by 13 publications

References 7 publications

A Hybrid Neural Autoencoder for Sensory Neuroprostheses and Its Applications in Bionic Vision

A Hybrid Neural Autoencoder for Sensory Neuroprostheses and Its Applications in Bionic Vision

Neuromorphic hardware for somatosensory neuroprostheses

Towards a Smart Bionic Eye: AI-powered artificial vision for the treatment of incurable blindness

Contact Info

Product

Resources

About