A computational model for task inference in visual search

Abstract

We develop a probabilistic framework to infer the ongoing task in visual search by revealing what the subject is looking for during a search process. Based on the level of difficulty, two types of tasks, easy and difficult, are investigated in this work, and individual models are customized for them according to their specific dynamics. We use Hidden Markov Models (HMMs) to serve as a model for the human cognitive process that is responsible for directing the center of gaze (COG) according to the task at hand during visual search and generating task-dependent eye trajectories. This generative model, then, is used to estimate the likelihood term in a Bayesian inference formulation to infer the task given the eye trajectory. In the easy task, focus of attention (FOA) often lands on targets, whereas in the difficult one, in addition to the on-target foci of attention, deployment of attention on nontarget objects happens very often. Therefore, we suggest a single-state and a multi-state HMM to serve as the cognitive process model of attention for the easy and difficult tasks, respectively.

Defending Yarbus: Eye movements reveal observers' task

“…Indeed, a large variety of studies has confirmed that eye movements contain rich signatures of the observer's mental task, including: predicting search target (Haji-Abolhassani & Clark, 2013;Rajashekar, Bovik, & Cormack, 2006;Zelinsky, Peng, & Samaras, 2013;Zelinsky, Zhang, & Samaras, 2008), predicting stimulus category (Borji, Tavakoli, Sihite, & Itti, 2013;Harel et al, 2009;O'Connell & Walther, 2012), predicting what number a person may randomly pick (Loetscher, Bockisch, Nicholls, & Brugger, 2010), predicting mental abstract tasks (Brandt & Stark, 1997;Ferguson & Breheny, 2011;Mast & Kosslyn, 2002;Meijering, van Rijn, Taatgen, & Verbrugge, 2012), predicting events (Bulling, Ward, Gellersen, & Tröster, 2011;Jang, Lee, Mallipeddi, Kwak, & Lee, 2011;Peters & Itti, 2007), classifying patients from controls (Jones & Klin, 2013;Tseng et al, 2012), and predicting driver's intent (Cyganek & Gruszczynski, 2014;Lethaus, Baumann, Köster, & Lemme, 2013). Several studies have investigated the role of eye movements in natural vision including: reading (Clark & O'Regan, 1998;Kaakinen & Hyönä, 2010;Rayner, 1979;Reichle, Rayner, & Pollatsek, 2003), visual search (Torralba, Oliva, Castelhano, & Henderson, 2006;Zelinsky, 2008), driving (Land & Lee, 1994;Land & Tatler, 2001), tea making (Land, Mennie, & Rusted, 1999), sandwich making (Hayhoe, Shrivastava, Mruczek, & Pelz, 2003), arithmetic and geometric problem solving …”

Advanced Statistical Methods for Eye Movement Analysis and Modelling: A Gentle Introduction

“…Fig. 37: Two solutions for the inverse Yarbus given a time series of observations: (1) the task-conditioned observable Markov model [54,45] (top panel) and the task-conditioned hidden Markov model [55] (bottom panel)…”

Prediction of search targets from fixations in open-world settings

“…typically fixations and saccades, in order to make predictions about user behaviour. For example, previous work has used visual behaviour analysis as a means to predict the users' tasks [1,3,13,17,22,41,42], visual activities [6,7,8,26,33], cognitive processes such as memory recall or high cognitive load [5,38], abstract thought processes [12,28], the type of a visual stimulus [4,10,23], interest for interactive image retrieval [11,15,20,25,32,37,43], which number a person has in mind [27], or -most recently -to predict the search target during visual search [2,16,34,41].…”