Improving Speech Intelligibility by Hearing Aid Eye-Gaze Steering: Conditions With Head Fixated in a Multitalker Environment

Abstract

The behavior of a person during a conversation typically involves both auditory and visual attention. Visual attention implies that the person directs his or her eye gaze toward the sound target of interest, and hence, detection of the gaze may provide a steering signal for future hearing aids. The steering could utilize a beamformer or the selection of a specific audio stream from a set of remote microphones. Previous studies have shown that eye gaze can be measured through electrooculography (EOG). To explore the precision and real-time feasibility of the methodology, seven hearing-impaired persons were tested, seated with their head fixed in front of three targets positioned at −30°, 0°, and +30° azimuth. Each target presented speech from the Danish DAT material, which was available for direct input to the hearing aid using head-related transfer functions. Speech intelligibility was measured in three conditions: a reference condition without any steering, a condition where eye gaze was estimated from EOG measures to select the desired audio stream, and an ideal condition with steering based on an eye-tracking camera. The “EOG-steering” improved the sentence correct score compared with the “no-steering” condition, although the performance was still significantly lower than the ideal condition with the eye-tracking camera. In conclusion, eye-gaze steering increases speech intelligibility, although real-time EOG-steering still requires improvements of the signal processing before it is feasible for implementation in a hearing aid.

Absolute Eye Gaze Estimation With Biosensors in Hearing Aids

“…In contrast, in a proof-of-concept study (Favre-Félix et al, 2018) recently demonstrated the potential of improving speech intelligibility in HI people in real time by assessing visual attention via EOG. Although the results of Favre-Félix et al (2018) provided first insights into visual attention steering via EOG, the study had substantial limitations regarding its potential value for applicability in HA technology. In that study, EOG was obtained from electrodes placed on the temples, which would not be feasible in real-life applications, where the sensors should preferably be embedded in the earmold.…”

“…While it has been shown that attended sound sources can be decoded from cortical brain responses via a correlation analysis of the scalp responses and an envelope representation of the attended input sound stream, a robust estimate requires several seconds of decoding, which is not applicable for real-time steering (Fuglsang et al, 2017;Dau et al, 2018). In contrast, in a proof-of-concept study (Favre-Félix et al, 2018) recently demonstrated the potential of improving speech intelligibility in HI people in real time by assessing visual attention via EOG. Although the results of Favre-Félix et al (2018) provided first insights into visual attention steering via EOG, the study had substantial limitations regarding its potential value for applicability in HA technology.…”

Head Motion Tracking Through in-Ear Wearables

“…Tracking head motion from only one ear might not give precise enough results, especially considering the absence of external reference points, such as a magnetometer. As having more fine grained data points enhances the precision of the estimation, we combined the streams of inertial data coming from the IMU sensors in the two earbuds 5 . We leverage the assumption that the two different IMUs are recording, from different vantage points, the same rotation.…”

“…Previous studies, especially from the medical community, highlight the importance of using directional hearing-aids leveraging the visual attention of the user [5,14]. For example, Favre-Félix et al [5] use electrooculography data (EOG) to characterize the visual attention of the patients. Their study shows how steering an hearing-aid by using EOG leads to better performance on the sentence correctness score.…”

“…However, because of the challenging signal processing required when dealing with it, the authors suggest that EOG may not be the best enabler to steer hearing-aids with a sufficiently high degree of precision. Head movements are closely linked to eye-movements [5], and therefore they are considered a good proxy to sense visual attention, too.…”