Activity recognition using eye-gaze movements and traditional interactions

Abstract

a b s t r a c tThe need for intelligent HCI has been reinforced by the increasing numbers of human-centered applications in our daily life. However, in order to respond adequately, intelligent applications must first interpret users' actions. Identifying the context in which users' interactions occur is an important step toward automatic interpretation of behavior. In order to address a part of this context-sensing problem, we propose a generic and application-independent framework for activity recognition of users interacting with a computer interface. Our approach uses Layered Hidden Markov Models (LHMM) and is based on eye-gaze movements along with keyboard and mouse interactions. The main contribution of the proposed framework is the ability to relate users' interactions to a task model in variant applications and for different monitoring purposes. Experimental results from two user studies show that our activity recognition technique is able to achieve good predictive accuracy with a relatively small amount of training data.

Individual user characteristics and information visualization

“…This research has typically focused on either identifying differences in gaze patterns for different visualizations [12], task types [17,26], and activities within a task [4], or on explaining differences in user accuracy between alternative visualization interfaces [22]. While these studies provide valuable insights on how different tasks and/or activities affect a user's gaze behaviors, they have traditionally ignored individual differences among study participants.…”

“…One approach to analyze eye tracking data is to apply data mining techniques, such as Hidden Markov Models [4], Scan-Path clustering [11], or specifically defined unsupervised algorithms [5,16]. While data mining methods can quickly identify clusters of similar attention patterns during visualization tasks, the results they return are often difficult to interpret, since unsupervised algorithms are typically applied as black-boxes.…”

Mouse Movement and Probabilistic Graphical Models Based E-Learning Activity Recognition Improvement Possibilistic Model

“…Various studies have been achieved for modeling and analysis of user's behavior during interaction with an online learning application using two of the most popular probabilistic graphical models: HMM and CRF. Courtemanche et al [27] used HMM approach to recognize the activity of learner within an e-learning platform based on mouse cursor interactions. In [28], HMM have been used for segmenting and labeling mouse trajectory into behavior acts such as cleaning, stopping and searching.…”

Gaze-based prediction of pen-based virtual interaction tasks

“…The first notable example is by Courtemanche et al (2011) who claim their approach to activity recognition to be the first one to incorporate eye movements. This work utilizes eye movements discretized in terms of interfacespecific AOIs in addition to keystrokes and mouse clicks input by the user during interaction.…”