Auditory Measures for the Next Billion Users

Abstract

A range of new technologies have the potential to help people, whether traditionally considered hearing impaired or not. These technologies include more sophisticated personal sound amplification products, as well as real-time speech enhancement and speech recognition. They can improve user’s communication abilities, but these new approaches require new ways to describe their success and allow engineers to optimize their properties. Speech recognition systems are often optimized using the word-error rate, but when the results are presented in real time, user interface issues become a lot more important than conventional measures of auditory performance. For example, there is a tradeoff between minimizing recognition time (latency) by quickly displaying results versus disturbing the user’s cognitive flow by rewriting the results on the screen when the recognizer later needs to change its decisions. This article describes current, new, and future directions for helping billions of people with their hearing. These new technologies bring auditory assistance to new users, especially to those in areas of the world without access to professional medical expertise. In the short term, audio enhancement technologies in inexpensive mobile forms, devices that are quickly becoming necessary to navigate all aspects of our lives, can bring better audio signals to many people. Alternatively, current speech recognition technology may obviate the need for audio amplification or enhancement at all and could be useful for listeners with normal hearing or with hearing loss. With new and dramatically better technology based on deep neural networks, speech enhancement improves the signal to noise ratio, and audio classifiers can recognize sounds in the user’s environment. Both use deep neural networks to improve a user’s experiences. Longer term, auditory attention decoding is expected to allow our devices to understand where a user is directing their attention and thus allow our devices to respond better to their needs. In all these cases, the technologies turn the hearing assistance problem on its head, and thus require new ways to measure their performance.

Harnessing the power of artificial intelligence to transform hearing healthcare and research

“…But the most promising use of AI in hearing devices is in replicating or enhancing functions that are normally performed by the auditory system 34 . By using DNNs to transform incoming sounds, AI could dramatically improve the signal processing in hearing devices.…”

Comparing In-ear EOG for Eye-Movement Estimation With Eye-Tracking: Accuracy, Calibration, and Speech Comprehension

“…Experiments conducted in less controlled environments that better represent the real world may introduce higher variability and unexpected effects in data, making analysis and interpretation of data more challenging. Sensing technologies can improve scene, situation, context, and intention awareness (Mehra et al, 2020 ; Slaney et al, 2020 ), and are promising tools for use in research with less experimental control. In hearing research, there is particularly a growing interest in sensing eye movement that is used as a metric to identify what sound (e.g., speech source) the listener is attending to.…”

Vocal and semantic cues for the segregation of long concurrent speech stimuli in diotic and dichotic listening—The Long-SWoRD test

“…Several studies, in particular, have found that when a listener is attending to one of two speech streams that correspond to two different voices, the cortical responses measured using intracortical recordings, electroencephalography (EEG), or magnetoencephalography (MEG) follow more strongly the temporal envelope of the attended stream than that of the unattended stream (Ding and Simon, 2012;Mesgarani and Chang, 2012;O'Sullivan et al, 2015). This process of uncovering which speech stream is being attended from the cortical temporal response function (TRF) is referred to as "attention decoding" (e.g., Biesmans et al, 2015)-a technique that potentially has important clinical applications (Slaney et al, 2020). In addition, the TRF can be used to unravel the neural correlates of the online processing of speech in situations in which sounds or voices compete with one another (Broderick et al, 2018;Di Liberto et al, 2015).…”