iCatcher+: Robust and Automated Annotation of Infants’ and Young Children’s Gaze Behavior From Videos Collected in Laboratory, Field, and Online Studies

Abstract

Technological advances in psychological research have enabled large-scale studies of human behavior and streamlined pipelines for automatic processing of data. However, studies of infants and children have not fully reaped these benefits because the behaviors of interest, such as gaze duration and direction, still have to be extracted from video through a laborious process of manual annotation, even when these data are collected online. Recent advances in computer vision raise the possibility of automated annotation of these video data. In this article, we built on a system for automatic gaze annotation in young children, iCatcher, by engineering improvements and then training and testing the system (referred to hereafter as iCatcher+) on three data sets with substantial video and participant variability (214 videos collected in U.S. lab and field sites, 143 videos collected in Senegal field sites, and 265 videos collected via webcams in homes; participant age range = 4 months–3.5 years). When trained on each of these data sets, iCatcher+ performed with near human-level accuracy on held-out videos on distinguishing “LEFT” versus “RIGHT” and “ON” versus “OFF” looking behavior across all data sets. This high performance was achieved at the level of individual frames, experimental trials, and study videos; held across participant demographics (e.g., age, race/ethnicity), participant behavior (e.g., movement, head position), and video characteristics (e.g., luminance); and generalized to a fourth, entirely held-out online data set. We close by discussing next steps required to fully automate the life cycle of online infant and child behavioral studies, representing a key step toward enabling robust and high-throughput developmental research.

An asynchronous, automated workflow for looking time experiments with infants

“…After data collection, data annotation is a second central bottleneck in extracting trial-level looking times. To automate this process, we used a recent model for automated gaze coding, iCatcher+, which is a neural network trained for coarse gaze classification from video (Erel et al, 2023).…”

“…To validate the ability of iCatcher+ to match human annotations in this new set-up, we replicated analyses from the original paper by Erel at al. (2023) on our own data.…”

“…A lookaway criterion could in principle lead to larger discrepancies than agreement over a fixed period as used in Erel et al (2023), because disagreement between humans and iCatcher+ about whether a lookaway was reached or not could lead to differences in the period over which looking time was calculated. In sum, iCatcher+ is a robust strategy for coding the duration of infant attention.…”

“…While these tools have been offered as general solutions to the problem of tracking gaze, automating the measurement of infant looking time in remote settings poses a set of specific challenges, such as detecting the infant among multiple faces in the scene, and correcting for movement, body position and variable camera angles. iCatcher+, the tool used in this paper, was designed to address these specific challenges, and has recently been reported to approach human-to-human reliability in annotating the gaze of infants in experimental studies (Erel et al, 2023). Importantly, it is not currently possible to asynchronously deliver experimental stimuli contingent on an infant's looking behavior.…”

“…How parents terminate trials introduces variability that a standardized experimenter-controlled design does not. Automated gaze coding via iCatcher+ has known failure modes in poor lighting conditions or unconventional poses (Erel et al, 2023). In the current experiment, 21 out of 135 videos could not be coded using iCatcher+ and so were dropped from the dataset (though some of these videos may have been excluded even if they were manually coded, e.g.…”