Vibrotactile Captioning of Musical Effects in Audio-Visual Media as an Alternative for Deaf and Hard of Hearing People: An EEG Study

Abstract

Standard captioning for the deaf and hard of hearing people cannot transmit the emotional information that music provides in support of the narrative in audiovisual media. We explore an alternative method using vibrotactile stimulation as a possible channel to transmit the emotional information contained in an audiovisual soundtrack and, thus, elicit a greater emotional reaction in hearing-impaired people. To achieve this objective, we applied two one-minute videos that were based on image sequences that were unassociated with dramatic action, maximizing the effect of the music and vibrotactile stimuli. While viewing the video, using EEG we recorded the brain activity of 9 female participants with normal hearing, and 7 female participants with very severe and profound hearing loss. The results show that the same brain areas are activated in participants with normal hearing watching the video with the soundtrack, and in participants with hearing loss watching the same video with a soft and rhythmic vibrotactile stimulation on the palm and fingertips, although in different hemispheres. These brain areas (auditory cortex, superior temporal cortex, medial frontal cortex, inferior frontal gyrus, superior temporal pole and Insula) have been consistently reported as areas involved in the emotional perception of music. We conclude that vibrotactile stimuli can generate cortex activation while watching audiovisual media in a similar way to sound. Thus, a further in-depth study of the possibilities of these stimuli can contribute to an alternative subtitling channel for enriching the audiovisual experience of hearing-impaired people.

Tecnologia Assistiva vibrotátil para a educação musical de surdos

“…Segundo Lucía et al (2020), as mesmas áreas cerebrais, embora em hemisférios diferentes, são ativadas em participantes ouvintes e em participantes surdos ao assistir ao mesmo vídeo com trilha sonora e com uma estimulação vibrotátil suave e rítmica na palma da mão e nas pontas dos dedos. Os autores concluíram que a Tecnologia Assistiva vibrotátil como um possível canal para transmitir as informações emocionais contidas em uma trilha sonora audiovisual e, com efeito, provocar uma maior reação emocional em surdos.…”

Escuchar con los ojos: el subtitulado para sordos de la música en el cine de terror

“…22-38 embargo, el sonido genera un número significativo de reacciones emocionales que no consiguen producir los subtítulos: si bien estos aumentan el nivel de atención necesario para procesar al mismo tiempo la información visual y verbal, no conllevan una mayor respuesta emocional. Además, existen otros estudios centrados en explorar una alternativa háptica para la evocación de las emociones transmitidas por la música para las personas con discapacidad auditiva (véanse Lucía Mulas et al, 2020;García López, 2022, Lucía Mulas, 2022. Sin embargo, estas investigaciones guardan menor relación con los objetivos de este estudio, por lo que nos permitiremos no ahondar en ellas.…”

Enhancing Musical Accessibility: A Novel Device for Individuals with Hearing Impairments using Vibrations and LED Lights Synchronized with Music Tempo

“…In 2010, computer science researchers and scientists came together to create a study and project similar to ours by using a vibrotactile stimulation to transmit emotional reactions in an audiovisual soundtrack for 2 minutes [5]. This experiment was conducted among a group of female participants from the University of Madrid, and out of all the volunteers, 9 were normal-hearing, 7 had hearing loss, and 4 were deaf.…”