Understanding the role of eye movement consistency in face recognition and autism through integrating deep neural networks and hidden Markov models

Abstract

Greater eyes-focused eye movement pattern during face recognition is associated with better performance in adults but not in children. We test the hypothesis that higher eye movement consistency across trials, instead of a greater eyes-focused pattern, predicts better performance in children since it reflects capacity in developing visual routines. We first simulated visual routine development through combining deep neural network and hidden Markov model that jointly learn perceptual representations and eye movement strategies for face recognition. The model accounted for the advantage of eyes-focused pattern in adults, and predicted that in children (partially trained models) consistency but not pattern of eye movements predicted recognition performance. This result was then verified with data from typically developing children. In addition, lower eye movement consistency in children was associated with autism diagnosis, particularly autistic traits in social skills. Thus, children’s face recognition involves visual routine development through social exposure, indexed by eye movement consistency.

Neural Correlates of the Advantage of Analytic Eye Movement Patterns in Face Recognition: An ERP Study

“…To better understand what cognitive processes were reflected in eye movement pattern, our multiple repression analysis showed that eye movement pattern could be predicted by a combination of the performance of Tower of London (number of moves) and P2 amplitude: a more eyes-focused pattern was predicted by better performance in Tower of London and smaller P2 amplitude. The relationship between Tower of London performance and eye movement pattern has been consistently reported in previous studies using EMHMM across visual tasks (e.g., Chan et al, 2018;Zheng et al, 2022;Hsiao et al, 2019), suggesting that eye movement patterns generally reflect individual differences in executive functions, particularly in planning and problem solving abilities (Unterrainer et al, 2004). The additional variance explained by P2 amplitude suggests that eye movement patterns reflect quality of perceptual representations in addition to executive function abilities.…”

“…These effects may be associated with the advantage of analytic eye movement pattern in face recognition performance. In addition, since previous research has shown that eye movement patterns are predictive of not only one's performance in a visual task but also cognitive abilities, particularly in executive function and visual attention abilities (e.g., Chan et al, 2018;Zheng, Ye, & Hsiao, 2022;Hsiao, Chan, Du, & Chan, 2019), we speculated that one's online eye movement behavior may reflect both high-level executive function activities and low-level perceptual representations as the result of the information extraction from the eye movements, making it an excellent predictor for recognition performance. Accordingly, here we also aimed to examine what factors, including eye movement pattern, cognitive ability, and ERP measures, could best account for individual differences in face recognition performance, and what factors, including cognitive ability and ERP measures, could best predict one's eye movement pattern.…”

The Application of Discrete Hidden Markov Model on Crosses of Diploid Plant

“…The hidden Markov model can help model pairs of events and their causes that cannot be observed directly [2]. This models have also been used in bioinformatics and biochemistry, some of them are [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13].…”

Visual attention to own‐ versus other‐race faces: Perspectives from learning mechanisms and task demands

“…Individuals learn to recognize faces by searching for and extracting relevant information for recognition and integrating the information to develop appropriate internal representations for efficient processing, which can be modulated by emotional and motivational factors. This learning mechanism can be simulated using a computational model that is designed to optimize recognition performance through joint learning of information extraction strategy (such as learning an effective visual routine) and efficient internal representations (such as developing both holistic and feature‐based face representations; see Hsiao et al (2022) for simulations using Deep Neural Network + Hidden Markov Model, or DNN + HMM, for face recognition). In this framework, both the perceptual expertise and social‐cognitive mechanisms can be understood as factors that modulate other‐race face processing given the relevant task demands.…”

“…Eye movement consistency reflects visual routine development: Eye movement consistency for own-race face recognition decreases gradually from early childhood into adulthood as a result of learning. Also, high eye movement consistency predicts better face recognition performance in children, whereas low eye movement consistency in face processing is associated with an autism diagnosis (Hsiao et al, 2022). As individuals may differ in their preference in adopting eyes-focused or nose-focused eye movement patterns for face processing (Chuk et al, 2014), eye movement consistency (i.e.…”