A deep hierarchy of predictions enables assignment of semantic roles in online speech comprehension

Abstract

Understanding speech requires mapping fleeting and often ambiguous soundwaves to meaning. Humans are known to exploit their capacity to contextualize to facilitate this process, but how internal knowledge is used and deployed in real time remains an open question. Existing models of speech processing focus on either word recognition irrespective of meaning or interactions among abstract linguistic representations without time constraints, providing only partial insights into the dynamics of speech comprehension. Here, we present a model that incrementally extracts multiple levels of information from continuous speech signals in real time, based on the inversion of a generative model that represents the listener’s internal knowledge of linguistic and non-linguistic processing levels in a nested temporal hierarchy. In each hierarchy, the model periodically incorporates bottom-up incoming evidence to update its internal representations and generate new top-down predictions. We show that a context level, beyond linguistic representations, can provide the model with semantic predictions informed by sensory inputs, crucial for the disambiguation among multiple meanings of the same word. We also show that hierarchical predictions can reduce peripheral processing effort via minimizing uncertainty and prediction error, especially when sensory precisions become degraded. With this proof-of-concept model we demonstrate that the deployment of hierarchical predictions is a possible strategy for the brain to utilize structured knowledge dynamically for speech comprehension.

Rhythmic modulation of prediction errors: A top-down gating role for the beta-range in speech processing

“…While Precoss-β, in contrast to most speech perception models [75][76][77], only covers the lower levels of the speech perception hierarchy and is simpler than contemporary ASR models [78,79], its main added feature is that it works on-line and potentially with low resources. Coupled with existing language models (e.g., GPT [80,81]), it could presumably reach high performance in on-line speech recognition.…”