Evaluating the Potential Gain of Auditory and Audiovisual Speech-Predictive Coding Using Deep Learning

Abstract

Sensory processing is increasingly conceived in a predictive framework in which neurons would constantly process the error signal resulting from the comparison of expected and observed stimuli. Surprisingly, few data exist on the accuracy of predictions that can be computed in real sensory scenes. Here, we focus on the sensory processing of auditory and audiovisual speech. We propose a set of computational models based on artificial neural networks (mixing deep feedforward and convolutional networks), which are trained to predict future audio observations from present and past audio or audiovisual observations (i.e., including lip movements). Those predictions exploit purely local phonetic regularities with no explicit call to higher linguistic levels. Experiments are conducted on the multispeaker LibriSpeech audio speech database (around 100 hours) and on the NTCD-TIMIT audiovisual speech database (around 7 hours). They appear to be efficient in a short temporal range (25–50 ms), predicting 50% to 75% of the variance of the incoming stimulus, which could result in potentially saving up to three-quarters of the processing power. Then they quickly decrease and almost vanish after 250 ms. Adding information on the lips slightly improves predictions, with a 5% to 10% increase in explained variance. Interestingly the visual gain vanishes more slowly, and the gain is maximum for a delay of 75 ms between image and predicted sound.

Modeling the initial state of early phonetic learning in infants

“…In this first experiment, we ask whether it is possible to induce 'innate' speech sound discrimination capabilities in our model and how the resulting initial state affects its developmental trajectory. Following Lavechin et al (2024), we chose a learning algorithm relying on auditory predictive coding at the core of the predictive brain hypothesis that has gained attention in the neuroscience community (Huang & Rao, 2011;Hueber et al, 2020). The algorithm learns by predicting future representations of audio based on present and past ones (see Methods).…”

Artificial neural networks to analyze and simulate language acquisition in children

“…Our learner model consists of two main components: 1) an acoustic model that incorporates a Contrastive Predictive Coding (CPC) algorithm followed by a K-means algorithm, which is in charge of learning discrete representations of the audio; and 2) a language model made of Long Short-Term Memory (LSTM) layers trained on the learned discrete units. The primary learning objective is to predict future audio observations from present and past ones, a process known as auditory predictive coding at the core of the predictive brain hypothesis that has attracted the attention of the neuroscience community (Barlow et al, 1961;Keller & Mrsic-Flogel, 2018;Hueber et al, 2020). As a statistical learning algorithm, the proposed model can provide us with insights about the aspects of language that can be acquired through statistical learning mechanisms applied to raw speech.…”

Standardized Judgment Method of Shooting Training Action Based on Digital Video Technology

“…Predictive coding is a technique to improve compression performance through statistical redundancy. Based on the previously encoded pixel values, the encoder can estimate and predict the pixel values to be encoded and decoded [26][27][28]. For a large number of static or slowly varying regions in the sequence image, the conditional patching method can be used to store the first frame image in the reference frame and send it to the other party.…”