Effects of training and using an audio-tactile sensory substitution device on speech-in-noise understanding

Abstract

Understanding speech in background noise is challenging. Wearing face-masks, imposed by the COVID19-pandemics, makes it even harder. We developed a multi-sensory setup, including a sensory substitution device (SSD) that can deliver speech simultaneously through audition and as vibrations on the fingertips. The vibrations correspond to low frequencies extracted from the speech input. We trained two groups of non-native English speakers in understanding distorted speech in noise. After a short session (30–45 min) of repeating sentences, with or without concurrent matching vibrations, we showed comparable mean group improvement of 14–16 dB in Speech Reception Threshold (SRT) in two test conditions, i.e., when the participants were asked to repeat sentences only from hearing and also when matching vibrations on fingertips were present. This is a very strong effect, if one considers that a 10 dB difference corresponds to doubling of the perceived loudness. The number of sentence repetitions needed for both types of training to complete the task was comparable. Meanwhile, the mean group SNR for the audio-tactile training (14.7 ± 8.7) was significantly lower (harder) than for the auditory training (23.9 ± 11.8), which indicates a potential facilitating effect of the added vibrations. In addition, both before and after training most of the participants (70–80%) showed better performance (by mean 4–6 dB) in speech-in-noise understanding when the audio sentences were accompanied with matching vibrations. This is the same magnitude of multisensory benefit that we reported, with no training at all, in our previous study using the same experimental procedures. After training, performance in this test condition was also best in both groups (SRT ~ 2 dB). The least significant effect of both training types was found in the third test condition, i.e. when participants were repeating sentences accompanied with non-matching tactile vibrations and the performance in this condition was also poorest after training. The results indicate that both types of training may remove some level of difficulty in sound perception, which might enable a more proper use of speech inputs delivered via vibrotactile stimulation. We discuss the implications of these novel findings with respect to basic science. In particular, we show that even in adulthood, i.e. long after the classical “critical periods” of development have passed, a new pairing between a certain computation (here, speech processing) and an atypical sensory modality (here, touch) can be established and trained, and that this process can be rapid and intuitive. We further present possible applications of our training program and the SSD for auditory rehabilitation in patients with hearing (and sight) deficits, as well as healthy individuals in suboptimal acoustic situations.

Training interoceptive awareness with real-time haptic vs. visual heartbeat feedback

“…Considering the efficacy of classical biofeedback in improving cardioception, we hypothesized that the training could be even more effective if it were to additionally incorporate principles of sensory augmentation. Previous findings coming from our and other groups indicate that the unisensory perceptual capabilities such as vision or hearing can be enhanced through the addition of supplementary, simultaneous feedback through additional senses (Abboud et al, 2014; Auvray et al, 2007; Bach-Y- Rita, 2004; Chebat et al, 2007; Cieśla et al, 2019, 2022; Gori et al, 2014; Haigh et al, 2013; Ptito, 2005). Furthermore, training with these sensory substitution and augmentation devices can induce fundamental organizational and developmental plastic changes in the brain (Abboud et al, 2014; Aggius-Vella et al, 2023; Striem-Amit & Amedi, 2014).…”

Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device

“…In recent years, this field has burgeoned, with several sensory substitution systems and algorithms currently being developed for various research aims. Most of these are based on the substitution of visual information through the auditory or tactile systems (Bach-y-Rita et al, 1969;Bach-y-Rita, 1983, 2004Ptito et al, 2005;Chebat et al, 2007) and the substitution of auditory information through the tactile system (Cieśla et al, 2019(Cieśla et al, , 2022, among others. Recent studies in our lab, both in the sighted (Netzer et al, 2021) and in the blind (Maimon et al, 2023), have also tested the ability to extend/augment visual-spatial perception (in both the front and the back) using auditory cues.…”

Effects of continuous tactile stimulation on auditory-evoked cortical responses depend on the audio-tactile phase

“…Only few studies have addressed this question, e.g., Fletcher and colleagues found that the presentation of speech-shaped tactile stimuli (time-varying stimuli resembling the waveform of auditory speech) can improve the intelligibility of simultaneously presented auditory speech in noise. This was observed in both normally-hearing listeners (Cieśla et al, 2022; Fletcher et al, 2018) and cochlear implant users (Fletcher et al, 2019; Huang et al, 2017) after they were trained on the audio-tactile speech. Similar studies with untrained listeners found partially conflicting results (Cieśla et al, 2019; Guilleminot & Reichenbach, 2022; Riecke et al, 2018; Riecke et al, 2019).…”