Auditory stimulus-response modeling with a match-mismatch task

Abstract

The relation between a continuous ongoing stimulus and the brain response that it evokes can be characterized by a stimulus-response model fit to the data. This systems-identification approach offers insight into perceptual processes within the brain, and it is also of potential practical use for devices such as Brain Computer Interfaces (BCI). The quality of the model can be quantified by measuring the fit with a regression problem, or by applying it to a classification task and measuring its performance. Here we focus on a match-mismatch task that entails deciding whether a segment of brain signal matches, via a model, the auditory stimulus that evoked it. The match-mismatch task can be used to compare performance of different stimulus-response models. We show that performance in a matchmismatch task and metrics summarizing regression accuracies can provide complementary insights in the relation between stimulus and response. Importantly, the match-mismatch task provides information about discriminatory power, making it directly applicable to BCI applications. Evaluation is performed on a freely available database, and code is available for scripts and functions to allow scrutiny of our results and facilitate comparative evaluation of future developments. 2.

Predicting speech intelligibility from EEG in a non-linear classification paradigm *

“…3) Discussion: The increase in performance by enlarging the input segment length is probably due to the model having more data to decide on. The same trend can be seen in auditory attention detection [14], [16] and in previous literature for CCA [6], [18]. Caution should be advised when using large input segment lengths (e.g., 20 seconds) as ceiling effects may occur due to some subjects obtaining the maximal score on the test set, at which point no further improvement can be gained.…”

“…Finally, the CCA model is evaluated using 2 different methods. In the first method, the classification is based on the difference of the distance between the transformed EEG and matched segment and the average distance between the transformed EEG and mismatched envelope segment, across trials per subject (as proposed by de Cheveigné et al [18]. In the second method, a single imposter segment is selected 1 second after the end of a matching segment (see Figure 1), following the same procedure as used for the dilated convolutional model and the decoder baseline model (see section II-B for more information).…”

“…If the training was stopped early, the model's weights were restored to their value in the epoch with the lowest validation loss. The CCA models were trained following the procedure for model G in de Cheveigné et al [18]. All experiments used input segment lengths of 10 seconds unless indicated otherwise.…”

Predicting speech intelligibility from EEG in a non-linear classification paradigm

“…Inspired by the recent developments in auditory attention decoding and CCA, we introduced a match/mismatch paradigm [17] in [18], based on [10], to relate an acoustic stimulus to a corresponding EEG recording. In this paradigm, a model with 3 inputs is presented: (a segment of) EEG, the speech stimulus envelope and an imposter envelope.…”

Modulation transfer functions for audiovisual speech

“…This implies that the model was optimized to predict AV correlations across speakers. The rCCA was trained using a match-mismatch scheme [ 55 ]. During cross-validation, rCCA models were trained on correctly matching video and audio data on four of the five folds, and correlations for each rCCA component were computed on the held-out validation fold.…”