Forecasting user attention during everyday mobile interactions using device-integrated and wearable sensors

Abstract

Visual attention is highly fragmented during mobile interactions, but the erratic nature of attention shifts currently limits attentive user interfaces to adapting after the fact, i.e. after shifts have already happened. We instead study attention forecasting -the challenging task of predicting users' gaze behaviour (overt visual attention) in the near future. We present a novel long-term dataset of everyday mobile phone interactions, continuously recorded from 20 participants engaged in common activities on a university campus over 4.5 hours each (more than 90 hours in total). We propose a proof-of-concept method that uses device-integrated sensors and body-worn cameras to encode rich information on device usage and users' visual scene. We demonstrate that our method can forecast bidirectional attention shifts and predict whether the primary attentional focus is on the handheld mobile device. We study the impact of different feature sets on performance and discuss the significant potential but also remaining challenges of forecasting user attention during mobile interactions.

Evaluation of Appearance-Based Methods and Implications for Gaze-Based Applications

“…Appearance-based methods are already suitable for applications only requiring measurement of relative changes in gaze direction over time, such as gaze-based user modelling, passive eye monitoring or detection of gaze patterns, such as smooth pursuit eye movements [13]. Also the latest attentive user interfaces could use appearance-based gaze estimation methods, such as for eye contact detection [42,54,70] or attention forecasting [56]. This suggests webcams could replace commercial eye trackers for some applications, and even enable new application scenarios such as online software-based services.…”

How Far Are We From Quantifying Visual Attention in Mobile HCI?

“…Recently, commercial eye trackers have become smaller and widely accessible which makes them good candidates for attention analysis [14]. Steil et al have used such an eye tracker together with mobile device-integrated sensors to forecast user attention [2]. While such dedicated systems bring us closer to the vision of pervasive attentive user interfaces, the fact that they require special-purpose equipment hinders large-scale deployment.…”

“…Consequently, actively managing user attention has emerged as a fundamental research challenge in human-computer interaction (HCI). With mobile devices being pervasively used in daily life, this challenge is even more relevant in mobile HCI where attentive behaviour has, as a result, become highly fragmented [1], [2]. A first step and key requirement to better understand and actively manage attention is to quantify when, how often, or for how long users visually attend to their devices.…”

“…Only few previous works have tried to quantify user attention and its allocation on mobile devices despite its fundamental importance for tasks such as predicting interruptibility [3] or modeling and understanding smartphone application usage [4]. Most of these studies required special-purpose eye tracking equipment [2] that constrained users' mobility or manual data annotation [1] that prevented the study of natural attentive behaviour at scale. One way to overcome these limitations is to instead rely on the high-resolution front-facing cameras readily integrated into these devices in combination with computer vision for gaze estimation [5], [6], [7], [8].…”

Providing Customised Information Panel Content Based on User Behavioral Patterns

“…The content personalization can be based on explicit user action (e.g. selection of topic or following a hashtag) or on analysis of user's location, previous actions [1,2]. Especially this second approach is crucial for most services.…”