Use of Computer Vision Analysis for Labeling Inattention Periods in Eeg Recordings With Visual Stimuli

Abstract

Electroencephalography (EEG) recordings with visual stimuli require detailed coding to determine the periods of participant’s attention. Here we propose to use a supervised machine learning model and off-the-shelf video cameras only. We extract computer vision-based features such as head pose, gaze, and face landmarks from the video of the participant, and train the machine learning model (multi-layer perceptron) on an initial dataset, then adapt it with a small subset of data from a new participant. Using a sample size of 23 autistic children, and training on additional 2560 labeled frames (equivalent to 85.3 seconds of the video) of a new participant, the median area under the receiver operating characteristic curve for inattention detection was 0.989 (IQR 0.984–0.993) and the median inter-rater reliability (Cohen’s kappa) with a trained human annotator was 0.888. Agreement with consensus annotation on four participants labeled independently by two human annotators was in the 0.827–0.960 range. Our results demonstrate the feasibility of automatic tools to detect inattention during EEG recordings, and its potential to reduce the subjectivity and time burden of human attention coding. The tool for model adaptation and visualization of the computer vision features is made publicly available to the research community.