Automated capture and delivery of assistive task guidance with an eyewear computer

Abstract

General rightsThis document is made available in accordance with publisher policies. Please cite only the published version using the reference above. ABSTRACTIn this paper we describe and evaluate an assistive mixed reality system that aims to augment users in tasks by combining automated and unsupervised information collection with minimally invasive video guides. The result is a fully self-contained system that we call GlaciAR (Glass-enabled Contextual Interactions for Augmented Reality). It operates by extracting contextual interactions from observing users performing actions. GlaciAR is able to i) automatically determine moments of relevance based on a head motion attention model, ii) automatically produce video guidance information, iii) trigger these guides based on an object detection method, iv) learn without supervision from observing multiple users and v) operate fully on-board a current eyewear computer (Google Glass). We describe the components of GlaciAR together with user evaluations on three tasks. We see this work as a first step toward scaling up the notoriously difficult authoring problem in guidance systems and an exploration of enhancing user natural abilities via minimally invasive visual cues.

Integration of Experts' and Beginners' Machine Operation Experiences to Obtain a Detailed Task Model

“…Many studies have investigated video-based user guidance in a variety of applications, for example, office work [2], [6], cooking [3]- [5], and farm work [7]. These systems provide guidance based on recorded expert experiences.…”

“…The automated acquisition of a task model from experts' behaviors has been investigated to reduce the burden of the manual collection of guidance data. Dima et al proposed a method of automatically integrating multiple experts' experiences and providing video guidance through a Google Glass [2], [6]. Chen et al [11] proposed a method for automatically extracting temporal interactions using hand shape and touch.…”

“…The general coordination is that the eye leads, followed by the head, and then the hand. Attention cues are used in the aforementioned works [2], [6] for guidance data acquisition. We expect that a comparison of gaze and at-tention among experts and beginners will provide detailed information about a task.…”

“…To measure the gaze, preceding studies have demonstrated strong correlations between gaze and head movement in an egocentric operation environment [6], [23]. We simply consider the average of the operator's initial gaze location as the reference of the attention location during the operation process, and use head movement to approximate the attention shift.…”

Detecting Transitions in Manual Tasks from Wearables: An Unsupervised Labeling Approach

“…A different environment is the kitchen where meal preparation could be of interest to health monitoring [71]. However, most aligned with the idea of this paper is the GlaciAR system [72], an authoring system for point-of-view videos based on object detection. We however employ wrist motion to detect significant steps.…”