An Eye Tracking Study on the Perception and Comprehension of Unimodal and Bimodal Linguistic Inputs by Deaf Adolescents

Abstract

An eye tracking experiment explored the gaze behavior of deaf individuals when perceiving language in spoken and sign language only, and in sign-supported speech (SSS). Participants were deaf (n = 25) and hearing (n = 25) Spanish adolescents. Deaf students were prelingually profoundly deaf individuals with cochlear implants (CIs) used by age 5 or earlier, or prelingually profoundly deaf native signers with deaf parents. The effectiveness of SSS has rarely been tested within the same group of children for discourse-level comprehension. Here, video-recorded texts, including spatial descriptions, were alternately transmitted in spoken language, sign language and SSS. The capacity of these communicative systems to equalize comprehension in deaf participants with that of spoken language in hearing participants was tested. Within-group analyses of deaf participants tested if the bimodal linguistic input of SSS favored discourse comprehension compared to unimodal languages. Deaf participants with CIs achieved equal comprehension to hearing controls in all communicative systems while deaf native signers with no CIs achieved equal comprehension to hearing participants if tested in their native sign language. Comprehension of SSS was not increased compared to spoken language, even when spatial information was communicated. Eye movements of deaf and hearing participants were tracked and data of dwell times spent looking at the face or body area of the sign model were analyzed. Within-group analyses focused on differences between native and non-native signers. Dwell times of hearing participants were equally distributed across upper and lower areas of the face while deaf participants mainly looked at the mouth area; this could enable information to be obtained from mouthings in sign language and from lip-reading in SSS and spoken language. Few fixations were directed toward the signs, although these were more frequent when spatial language was transmitted. Both native and non-native signers looked mainly at the face when perceiving sign language, although non-native signers looked significantly more at the body than native signers. This distribution of gaze fixations suggested that deaf individuals – particularly native signers – mainly perceived signs through peripheral vision.

Audio-visual integration in noise: Influence of auditory and visual stimulus degradation on eye movements and perception of the McGurk effect

“…In conjunction with this, CI users perceive the McGurk effect more often compared to normal hearing listeners (Stropahl, Schellhardt, and Debener, 2017 ). This benefit of visual information and increased perception of the McGurk effect could be due to CI users’ tendency to look at the mouth more compared to people with normal hearing (Mastrantuono, Saldaña & Rodríguez-Ortiz, 2017 ). People with CIs might look at the mouth more in order to help them get more information from the visual signal, in the face of auditory degradation.…”

Sign language experience redistributes attentional resources to the inferior visual field

“…Eye gaze tracking studies have observed that, during SL communication, signers maintain their gaze on the face of their interlocutor rather that looking at the hands of that individual (Agrafiotis, Canagarajah, Bull, & Dye, 2003;Emmorey, Korpics, & Petronio, 2009;Mastrantuono, Saldaña, & Rodríguez-Ortiz, 2017;Muir & Richardson, 2005). Subtle visual (phonological) distinctions (Siple, 1978) as well as grammatical markings (Brentari & Crossley, 2002;Elliott & Jacobs, 2013;Grossman & Kegl, 2007) that occur on the face of the signer, likely require signers to foveate the face to maintain the high levels of visual acuity required.…”

The Role of Multiple Articulatory Channels of Sign-Supported Speech Revealed by Visual Processing

“…The benefits of a proper sign language model from birth is especially demonstrated by studies on deaf children with cochlear implants born from deaf parents. These children, exposed to an intact and fluent model of sign language, either American (Davidson, Lillo-Martin, & Pichler, 2014), Spanish (Mastrantuono, Saldaña, & Rodríguez-Ortiz, 2017) or Italian (Rinaldi & Caselli, 2014), develop age-appropriate spoken language skills.. The exposure to an intact model of sign language from birth also benefits the development of executive functions, as tested for French Sign Language (Courtin, 2000) and American Sign Language (Dye & Hauser, 2014;Hall, Eigsti, Bortfeld, & Lillo-Martin, 2017b;Schick, de Villiers, de Villiers, & Hoffmeister, 2007).…”

“…Studies in that field indicate that deaf observers mainly foveate the face area. Deaf perceivers, both cochlear implant users and native signers, have been observed to look at the face more than 95% of the time when attending sign language (Spanish Sign Language), as well as when attending spoken language and SSS (Mastrantuono et al, 2017).…”

“…Mouthing was estimated to accompany 69% of signs of British Sign Language (Sutton-Spence, 2007) and 64% of signs of Spanish Sign Language stories used in a recent study (Mastrantuono et al, 2017). It is assumed that sign perception while looking at the face area involves covert attention.…”