A case study in phenomenology of visual experience with retinal prosthesis versus visual-to-auditory sensory substitution

Maimon, Amber; Yizhar, Or; Buchs, Galit; Heimler, Benedetta; Amedi, Amir

doi:10.1016/j.neuropsychologia.2022.108305

Cited by 9 publications

(4 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, Witzel et al (2022) published a study exploring the automaticity of novel perceptual experiences by employing a sensory augmentation device for perceiving the north direction. These studies further support the subjective reports presented in case studies that indicate acquired automaticity and transparency following extensive use of sensory substitution devices (Ward and Meijer, 2010;Maimon et al, 2022).…”

Section: Discussionsupporting

confidence: 85%

Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device

Shvadron

Snir²,

Maimon³

et al. 2023

Front. Hum. Neurosci.

Self Cite

View full text Add to dashboard Cite

Current advancements in both technology and science allow us to manipulate our sensory modalities in new and unexpected ways. In the present study, we explore the potential of expanding what we perceive through our natural senses by utilizing a visual-to-auditory sensory substitution device (SSD), the EyeMusic, an algorithm that converts images to sound. The EyeMusic was initially developed to allow blind individuals to create a spatial representation of information arriving from a video feed at a slow sampling rate. In this study, we aimed to use the EyeMusic for the blind areas of sighted individuals. We use it in this initial proof-of-concept study to test the ability of sighted subjects to combine visual information with surrounding auditory sonification representing visual information. Participants in this study were tasked with recognizing and adequately placing the stimuli, using sound to represent the areas outside the standard human visual field. As such, the participants were asked to report shapes’ identities as well as their spatial orientation (front/right/back/left), requiring combined visual (90° frontal) and auditory input (the remaining 270°) for the successful performance of the task (content in both vision and audition was presented in a sweeping clockwise motion around the participant). We found that participants were successful at a highly above chance level after a brief 1-h-long session of online training and one on-site training session of an average of 20 min. They could even draw a 2D representation of this image in some cases. Participants could also generalize, recognizing new shapes they were not explicitly trained on. Our findings provide an initial proof of concept indicating that sensory augmentation devices and techniques can potentially be used in combination with natural sensory information in order to expand the natural fields of sensory perception.

show abstract

Section: Discussionsupporting

confidence: 85%

Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device

Shvadron

Snir²,

Maimon³

et al. 2023

Front. Hum. Neurosci.

Self Cite

View full text Add to dashboard Cite

show abstract

“…The incorporation of artificial intelligence could open a wealth of possibilities for the blind and visually impaired with respect to providing them with freedom and independence in unfamiliar or changing environments. Artificial intelligence is already being integrated into rehabilitative systems for the sensory impaired, for example retinal prostheses ( Barnes, 2012 ; Weiland et al, 2012 ) and hearing aids ( Crowson et al, 2020 ; Lesica et al, 2021 ), yet SSD systems hold the potential for providing a more transparent and automatic perceptual experience ( Ward and Meijer, 2010 ; Maimon et al, 2022 ) and therefore could be particularly powerful when combined with real time computer vision.…”

Section: Discussionmentioning

confidence: 99%

The Topo-Speech sensory substitution system as a method of conveying spatial information to the blind and vision impaired

Maimon

Wald

et al. 2023

Front. Hum. Neurosci.

Self Cite

View full text Add to dashboard Cite

Humans, like most animals, integrate sensory input in the brain from different sensory modalities. Yet humans are distinct in their ability to grasp symbolic input, which is interpreted into a cognitive mental representation of the world. This representation merges with external sensory input, providing modality integration of a different sort. This study evaluates the Topo-Speech algorithm in the blind and visually impaired. The system provides spatial information about the external world by applying sensory substitution alongside symbolic representations in a manner that corresponds with the unique way our brains acquire and process information. This is done by conveying spatial information, customarily acquired through vision, through the auditory channel, in a combination of sensory (auditory) features and symbolic language (named/spoken) features. The Topo-Speech sweeps the visual scene or image and represents objects’ identity by employing naming in a spoken word and simultaneously conveying the objects’ location by mapping the x-axis of the visual scene or image to the time it is announced and the y-axis by mapping the location to the pitch of the voice. This proof of concept study primarily explores the practical applicability of this approach in 22 visually impaired and blind individuals. The findings showed that individuals from both populations could effectively interpret and use the algorithm after a single training session. The blind showed an accuracy of 74.45%, while the visually impaired had an average accuracy of 72.74%. These results are comparable to those of the sighted, as shown in previous research, with all participants above chance level. As such, we demonstrate practically how aspects of spatial information can be transmitted through non-visual channels. To complement the findings, we weigh in on debates concerning models of spatial knowledge (the persistent, cumulative, or convergent models) and the capacity for spatial representation in the blind. We suggest the present study’s findings support the convergence model and the scenario that posits the blind are capable of some aspects of spatial representation as depicted by the algorithm comparable to those of the sighted. Finally, we present possible future developments, implementations, and use cases for the system as an aid for the blind and visually impaired.

show abstract

“…This approach might be a promising rehabilitative venue to further tune face classification abilities developed via natural experience in visually restored patients ( Gandhi et al, 2017 ). We suggest that such multisensory training approach might maximize the restoration outcomes, as the familiar sense (e.g., audition) might guide the restored sense (e.g., vision) to recruit its typical sensory cortex by promoting a network adaptability process ( Heimler et al, 2015 ; Heimler and Amedi, 2020 ; Maimon et al, 2022 ). Future studies may more systematically investigate this intriguing hypothesis and track the extent to which such types of training might indeed aid the (re) establishment of typical cortical recruitment by the restored visual input, in line with the predictions of the computational-selective cortical organization.…”

Section: Discussionmentioning

confidence: 99%

Face shape processing via visual-to-auditory sensory substitution activates regions within the face processing networks in the absence of visual experience

2022

Self Cite

View full text Add to dashboard Cite

Previous evidence suggests that visual experience is crucial for the emergence and tuning of the typical neural system for face recognition. To challenge this conclusion, we trained congenitally blind adults to recognize faces via visual-to-auditory sensory-substitution (SDD). Our results showed a preference for trained faces over other SSD-conveyed visual categories in the fusiform gyrus and in other known face-responsive-regions of the deprived ventral visual stream. We also observed a parametric modulation in the same cortical regions, for face orientation (upright vs. inverted) and face novelty (trained vs. untrained). Our results strengthen the conclusion that there is a predisposition for sensory-independent and computation-specific processing in specific cortical regions that can be retained in life-long sensory deprivation, independently of previous perceptual experience. They also highlight that if the right training is provided, such cortical preference maintains its tuning to what were considered visual-specific face features.

show abstract

A case study in phenomenology of visual experience with retinal prosthesis versus visual-to-auditory sensory substitution

Cited by 9 publications

References 62 publications

Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device

Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device

The Topo-Speech sensory substitution system as a method of conveying spatial information to the blind and vision impaired

Face shape processing via visual-to-auditory sensory substitution activates regions within the face processing networks in the absence of visual experience

Contact Info

Product

Resources

About