A self-training program for sensory substitution devices

Buchs, Galit; Haimler, Benedetta; Kerem, Menachem; Maidenbaum, Shachar; Braun, Liraz; Amedi, Amir

doi:10.1371/journal.pone.0250281

Cited by 9 publications

(9 citation statements)

References 72 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our findings indicate that sighted participants can indeed process spatial information starting in their visual field (frontal 90 • ) and continuing in their auditory field (the remaining 270 • ) to create a unified image. In addition, this study replicated the results of a previous study conducted by Buchs et al (2021), showing the efficacy of online training for visual-to-auditory SSDs. We show that subjects could even draw the stimuli within a short period of time, to some success when performing a task demanding the conversion of the entire surrounding image into a 2D visual rendering of the space (combining the back, front, and both sides onto a single 2D plane).…”

Section: Discussionsupporting

confidence: 87%

See 1 more Smart Citation

Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device

Shvadron

Snir²,

Maimon³

et al. 2023

Front. Hum. Neurosci.

Self Cite

View full text Add to dashboard Cite

Current advancements in both technology and science allow us to manipulate our sensory modalities in new and unexpected ways. In the present study, we explore the potential of expanding what we perceive through our natural senses by utilizing a visual-to-auditory sensory substitution device (SSD), the EyeMusic, an algorithm that converts images to sound. The EyeMusic was initially developed to allow blind individuals to create a spatial representation of information arriving from a video feed at a slow sampling rate. In this study, we aimed to use the EyeMusic for the blind areas of sighted individuals. We use it in this initial proof-of-concept study to test the ability of sighted subjects to combine visual information with surrounding auditory sonification representing visual information. Participants in this study were tasked with recognizing and adequately placing the stimuli, using sound to represent the areas outside the standard human visual field. As such, the participants were asked to report shapes’ identities as well as their spatial orientation (front/right/back/left), requiring combined visual (90° frontal) and auditory input (the remaining 270°) for the successful performance of the task (content in both vision and audition was presented in a sweeping clockwise motion around the participant). We found that participants were successful at a highly above chance level after a brief 1-h-long session of online training and one on-site training session of an average of 20 min. They could even draw a 2D representation of this image in some cases. Participants could also generalize, recognizing new shapes they were not explicitly trained on. Our findings provide an initial proof of concept indicating that sensory augmentation devices and techniques can potentially be used in combination with natural sensory information in order to expand the natural fields of sensory perception.

show abstract

Section: Discussionsupporting

confidence: 87%

“…The phenomenological questionnaire was based on a questionnaire from the study by Buchs et al (2021). It included questions regarding the perceived learning and perceived difficulty in each experiment stage and the pleasantness of the stimuli.…”

Section: Questionnairementioning

confidence: 99%

Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device

Shvadron

Snir²,

Maimon³

et al. 2023

Front. Hum. Neurosci.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Other familiarization methods have been studied and have shown improvements in the use of SSDs. For example, prior to the experimental task, some studies simultaneously displayed to participants an image and its equivalent soundscape (Ambard et al, 2015 ; Buchs et al, 2021 ). In another study (Auvray et al, 2007 ), participants were enrolled in an intensive training of 3 h. Using only a verbal explanation of the visual-to-auditory encoding scheme as been shown to be efficient to understand the main principles of the encoding scheme (Kim and Zatorre, 2008 ; Buchs et al, 2021 ; Scalvini et al, 2022 ).…”

Section: Discussionmentioning

confidence: 99%

“…For example, prior to the experimental task, some studies simultaneously displayed to participants an image and its equivalent soundscape (Ambard et al, 2015 ; Buchs et al, 2021 ). In another study (Auvray et al, 2007 ), participants were enrolled in an intensive training of 3 h. Using only a verbal explanation of the visual-to-auditory encoding scheme as been shown to be efficient to understand the main principles of the encoding scheme (Kim and Zatorre, 2008 ; Buchs et al, 2021 ; Scalvini et al, 2022 ). The aim of the current study was not to directly investigate the effect of a short and active familiarization method on localization performance but it shows that a short practice might be sufficient to acquire the sensorimotor contingencies.…”

Section: Discussionmentioning

confidence: 99%

Cross-modal correspondence enhances elevation localization in visual-to-auditory sensory substitution

et al. 2023

View full text Add to dashboard Cite

IntroductionVisual-to-auditory sensory substitution devices are assistive devices for the blind that convert visual images into auditory images (or soundscapes) by mapping visual features with acoustic cues. To convey spatial information with sounds, several sensory substitution devices use a Virtual Acoustic Space (VAS) using Head Related Transfer Functions (HRTFs) to synthesize natural acoustic cues used for sound localization. However, the perception of the elevation is known to be inaccurate with generic spatialization since it is based on notches in the audio spectrum that are specific to each individual. Another method used to convey elevation information is based on the audiovisual cross-modal correspondence between pitch and visual elevation. The main drawback of this second method is caused by the limitation of the ability to perceive elevation through HRTFs due to the spectral narrowband of the sounds.MethodIn this study we compared the early ability to localize objects with a visual-to-auditory sensory substitution device where elevation is either conveyed using a spatialization-based only method (Noise encoding) or using pitch-based methods with different spectral complexities (Monotonic and Harmonic encodings). Thirty eight blindfolded participants had to localize a virtual target using soundscapes before and after having been familiarized with the visual-to-auditory encodings.ResultsParticipants were more accurate to localize elevation with pitch-based encodings than with the spatialization-based only method. Only slight differences in azimuth localization performance were found between the encodings.DiscussionThis study suggests the intuitiveness of a pitch-based encoding with a facilitation effect of the cross-modal correspondence when a non-individualized sound spatialization is used.

show abstract

“…However, the acquisition of skilled use of a device does not necessarily imply the development of a genuine perceptual feel (Arnold et al, 2017; Auvray & Harris, 2014; Block, 2003; Brown et al, 2011; Deroy & Auvray, 2012). Evidence for the effects of sensory substitution or augmentation consists mainly of performance in very specific discrimination tasks with small sets of stimuli (e.g., Auvray et al, 2007; Bach-y-Rita et al, 1969; Buchs et al, 2021; Chebat et al, 2015; Díaz et al, 2012; Favela et al, 2018; Goeke et al, 2016; Haigh et al, 2013; Lobo et al, 2014; Proulx et al, 2015; Proulx et al, 2008; Travieso et al, 2015). Yet observed performance in this kind of discrimination may be explained by explicit cognitive interpretation rather than by the development of a perceptual feel (Deroy & Auvray, 2012; see also Goeke et al, 2016; Schumann & O'Regan, 2017).…”

Section: Introductionmentioning

confidence: 99%

Can perception be extended to a “feel of north”? Tests of automaticity with the NaviEar

et al. 2022

View full text Add to dashboard Cite

This study investigated the potential for the development of novel perceptual experiences through sustained training with a sensory augmentation device. We developed (1) a new geomagnetic sensory augmentation device, the NaviEar, and (2) a battery of tests for automaticity in the use of the device. The NaviEar translates head direction toward north into continuous sound according to a “wind coding” principle. To facilitate automatization of use, its design is informed by considerations of the embodiment of spatial orientation and multi-sensory integration, and it uses a sensory coding scheme derived from means for auditory perception of wind direction that is common in sailing because it is easy to understand and use. The test battery assesses different effects of automaticity (interference, rigidity of responses, and dynamic integration) assuming that automaticity is a necessary criterion to show the emergence of perceptual feel, that is, an augmented experience with perceptual phenomenal quality. We measured performance in simple training tasks, administered the tests for automaticity, and assessed subjective reports through a questionnaire. Results suggest that the NaviEar is easy and comfortable to use and has a potential for applications in real-world situations. Despite high usability, however, a 5-day training with the NaviEar did not reach levels of automaticity that are indicative of perceptual feel. We propose that the test battery for automaticity may be used as a benchmark test for iterative research on perceptual experiences in sensory augmentation and sensory substitution.

show abstract

A self-training program for sensory substitution devices

Cited by 9 publications

References 72 publications

Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device

Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device

Cross-modal correspondence enhances elevation localization in visual-to-auditory sensory substitution

Can perception be extended to a “feel of north”? Tests of automaticity with the NaviEar

Contact Info

Product

Resources

About