Listening effort is not the same as speech intelligibility score

Abstract

Speech perception and listening effort are complicated and interrelated concepts. One might assume that intelligibility performance (percent correct) is a proxy for listening effort, but there are some reasons to challenge whether that is actually true. Correct responses in speech perception tests could reflect effortful mental processing, and a completely wrong answer could evoke very little effort, especially if the misperception itself is linguistically well-formed and sensible. This paper presents evidence that listening effort is not a function of the proportion of words correct, but is rather driven by the types of errors, position of errors within a sentence, and the need to resolve ambiguity, reflecting how easily the listener can make sense of a perception. We offer a taxonomy of error types that is both intuitive and also consistent with data from two experiments measuring listening effort with careful controls to either elicit specific kinds of mistakes or to track specific mistakes retrospectively. Participants included individuals with normal hearing or with cochlear implants. In two experiments of sentence repetition, listening effort – indexed by changes in pupil size – was found to scale with the amount of perceptual restoration needed (phoneme versus whole word), and also scale with the sensibility of responses, but not with the number of intelligibility errors. Although mental corrective action and number of mistakes can scale together in many experiments, it is possible to dissociate them in order to advance toward a more explanatory (rather than correlational) account of listening effort.

Cognitive and Physiological Measures of Listening Effort During Degraded Speech Perception: Relating Dual-Task and Pupillometry Paradigms

“…That is, engaging more effort did not mean an individual would be more successful recognizing words. This is consistent with several previous studies that find that intelligibility is separable from effort (Mackersie & Cones, 2011;Sarampalis et al, 2009;Winn & Teece, 2020). It is assumed that extra cognitive resources are engaged to compensate for challenging listening situations, but this extra effort seems to hit a ceiling of effectiveness for a listener's speech accuracy.…”

“…This seems to imply that after accounting for overall difficulty, subjects who exert more effort do not show better accuracy. There is, in fact, a growing body of evidence that speech accuracy and listening effort are not inherently tied together and that increases in effort do not necessitate improvements in performance (Mackersie & Cones, 2011;Sarampalis et al, 2009;Winn & Teece, 2020).…”

The costs (and benefits) of effortful listening on context processing: A simultaneous electrophysiology, pupillometry, and behavioral study

“…This is important because, while acoustic challenge increases cognitive demand, it is not the only factor determining effortful listening. Rather, listening effort reflects the deliberate allocation of cognitive and neural resources in response to increased acoustic challenge, which can vary substantially within a given listening situation (Zekveld et al, 2010(Zekveld et al, , 2018Winn & Teece, 2021).…”