Statistically Optimal Multisensory Cue Integration: A Practical Tutorial

Abstract

Humans combine redundant multisensory estimates into a coherent multimodal percept. Experiments in cue integration have shown for many modality pairs and perceptual tasks that multisensory information is fused in a statistically optimal manner: observers take the unimodal sensory reliability into consideration when performing perceptual judgments. They combine the senses according to the rules of Maximum Likelihood Estimation to maximize overall perceptual precision. This tutorial explains in an accessible manner how to design optimal cue integration experiments and how to analyse the results from these experiments to test whether humans follow the predictions of the optimal cue integration model. The tutorial is meant for novices in multisensory integration and requires very little training in formal models and psychophysical methods. For each step in the experimental design and analysis, rules of thumb and practical examples are provided. We also publish Matlab code for an example experiment on cue integration and a Matlab toolbox for data analysis that accompanies the tutorial online. This way, readers can learn about the techniques by trying them out themselves. We hope to provide readers with the tools necessary to design their own experiments on optimal cue integration and enable them to take part in explaining when, why and how humans combine multisensory information optimally.

Sensory Cue Combination in Children Under 10 Years of Age

“…proportion not greater than the expected Type I error rate. However, it remains the obvious alternative to existing theory Ernst, Rohde, & van Dam, 2016). Our central approach is to see if a new task can provide consistent evidence of cue combination in this age range.…”

“…To show where they perceived Piccolina to be, children used a virtual tool that they controlled using a computer mouse. This allows us to analyse the data with standard statistical tests to compare unimodal precision and bimodal precision (Ernst et al, 2016).…”

Neural Correlates of Multisensory Reliability and Perceptual Weights Emerge at Early Latencies during Audio-visual Integration

“…By doing so, more reliable cues are assigned a higher weight and have stronger influence on the perceptual estimate. In most cases, this leads to a more precise and reliable percept Ernst, 2006;Ernst and Bülthoff, 2004;Fetsch et al, 2013;Rohde et al, 2015].…”

Mechanism of integrating force and vibrotactile cues for 3D user interaction within virtual environments

“…This agreed with the observations on tactile cues for sensing 3D surfaces [11], establishing a baseline for examining the integration of force and vibrotactile cues. The integration of both cues was examined on Acc by using MLE [14], because TCT was non-differentiable. Each testing block yielded an experimentally estimated Gaussian distribution, according to the MLE's rules.…”

“…Proper integration of both cues is therefore paramount for creating intuitive VEs and facilitating 3D interaction. Existing reports on multi-sensory integration have focused on cues of the visual and haptic modalities [5,14], disparity and texture cues of the visual modality [8], and force and position cues of the haptic modality [4]. These integrations had collocated cues and followed maximum likelihood estimation (MLE) [5].…”