Just-in-time: gaze guidance in natural behavior

Abstract

Eye movements in the natural environment have primarily been studied for over-learned everyday activities such as tea-making, sandwich making, driving that have a fixed sequence of actions associated with them. These studies indicate a just-in-time strategy of fixations i.e. the fixation provides the information for a particular action immediately precedes that action. However, it is unclear if this strategy is also in play when the task is novel and a sequence of actions must be planned in the moment. To study attention mechanisms in a novel task in a natural environment, we recorded gaze and body movement data in a virtual environment while subjects performed a sorting task where they sorted objects based on object features on a life-size shelf. To study the action planning and execution related gaze guidance behavior we also controlled the complexity of the sorting task by introducing EASY and HARD tasks. We show that subjects are close to optimal while performing EASY trials and are more sub-optimal while performing HARD tasks. Based on the scan-paths as well as latency of first fixations on the task-related ROIs during action planning and execution we show that subjects use a just-in-time strategy of fixating on the task relevant objects. From our findings, we can conclude that subjects use the just-in-time strategy in a way that sacrifices optimality by offloading cognitive task demands on the environment. These findings also lend further support to the embodied cognitive framework of cognitive processing in natural environments.

Low-Dimensional Representations of Visuomotor Coordination in Natural Behavior

“…These findings have shown eye movements gather information from the environment proactively and in anticipation of upcoming manual action (Land and Furneaux, 1997;Land and Hayhoe, 2001). Keshava et al (2024) showed that vision-for-action in naturalistic tasks can be accomplished with just-in-time representations where gaze fixations relay information for an action right before the action. Common actions, such as picking up objects, require only intermittent visual fixations to guide actions rapidly and efficiently.…”

Action Affordance Affects Proximal and Distal Goal-Oriented Planning

“…In virtual reality (VR), realistic actions can be studied by simulating an interaction within an environment. In these simulations, using interfaces such as VR controllers, ego-centric visual feedback of a body part, most often a hand and sometimes the whole body (Keshava et al, 2021) can be designed. These interfaces usually consist of a hand-held device that can be controlled the index finger or the thumb alone.…”