Action Affordance Affects Proximal and Distal Goal-Oriented Planning

Abstract

Seminal studies on human cognitive behavior have been conducted in controlled laboratory settings, demonstrating that visual attention is mainly goal-directed and allocated based on the action performed. However, it is unclear how far these results generalize to cognition in more naturalistic settings. The present study investigates active inference processes revealed by eye movements during interaction with familiar and novel tools with two levels of realism of the performed action. We presented participants with 3D tool models that were either familiar or unfamiliar, oriented congruent or incongruent to their handedness, and asked participants to interact with them by either lifting or using. Importantly, we used the same experimental design in two setups. In the first experiment, participants interacted with a VR controller; in the second, they performed the task with a more realistic interaction setup that allowed differentiated hand and finger movements. We used linear mixed models to determine the odds of fixations on the tool effector vs. handle before action initiation. The results show that participants fixate more on the tool's effector part before action initiation during the use task for unfamiliar tools. Furthermore, under more natural interaction conditions, subjects fixate more on the tool's handle as a function of the tool's orientation, well before the action was executed. Secondly, the spatial viewing bias on the tool reveals early fixations are associated with the task and familiarity of the tools. In contrast, later fixations are associated with the manual planning of the interaction. In sum, the findings from the experiments suggest that fixations are made in a task-oriented way to extract the mechanical properties of the tools to plan the intended action well before action initiation. Further, in more natural conditions, fixations are made towards proximal goals of optimally planning the grasp even when the perceived action on the tools is identical. Thus, the realism of the action in a virtual environment seems to be as important as the sensory stimulation.

Combining EEG and Eye-Tracking in Virtual Reality - Obtaining Fixation-Onset ERPs and ERSPs

“…Using our eye movement classification algorithm, we showed that we could accurately classify eye movements of three-dimensional free-exploration data and that we can generate fERPs and fERSPs, proving that combining EEG and free-viewing virtual reality setups is possible. We investigated the classification quality using our modified version of a velocity-based classification algorithm (Dar et al, 2021;Keshava et al, 2023;Voloh et al, 2020), correcting for subject movement in the virtual environment. Furthermore, we compared two data-segmentation methods dealing with varying noise levels across a long recording.…”

“…We, therefore, introduced a movement correction before calculating the eye angular velocity (Keshava et al, 2023) used to define gazes and saccades. The key to our approach is first to compute the hit point's movement in allocentric coordinates and then translate it to the required change of eye direction in allocentric coordinates.…”

“…The calculated angular velocities (Keshava et al, 2023) were classified into gaze events (smooth pursuit and fixations) and saccades based on an adaptively calculated threshold (Voloh et al, 2020). This experiment did not have predefined trials, so applying the adaptive threshold method to the entire dataset would only detect very big saccades.…”

“…We created a virtual, three-dimensional environment for our participants to explore while recording eye-tracking and EEG. Eye movements were classified using a velocity-based algorithm and an adaptively determined velocity threshold (Keshava et al, 2023;Voloh et al, 2020).…”

Just-in-time: gaze guidance in natural behavior

“…In recent years, virtual reality (VR) and mobile sensing have offered great opportunity to create controlled, natural environments. Here, subjects' eye and body movements can be measured reliably along with their interactions with the environment (Keshava et al, 2020(Keshava et al, , 2021Clay et al, 2019;Mann et al, 2019). Experiments in virtual environments have grown popular in recent years and have shown promise towards studying cognition in naturalistic and controlled environments.…”