Ajalon: Simplifying the authoring of wearable cognitive assistants

Abstract

Summary Wearable Cognitive Assistance (WCA) amplifies human cognition in real time through a wearable device and low‐latency wireless access to edge computing infrastructure. It is inspired by, and broadens, the metaphor of GPS navigation tools that provide real‐time step‐by‐step guidance, with prompt error detection and correction. WCA applications are likely to be transformative in education, health care, industrial troubleshooting, manufacturing, assisted driving, and sports training. Today, WCA application development is difficult and slow, requiring skills in areas such as machine learning and computer vision that are not widespread among software developers. This paper describes Ajalon, an authoring toolchain for WCA applications that reduces the skill and effort needed at each step of the development pipeline. Our evaluation shows that Ajalon significantly reduces the effort needed to create new WCA applications.

ARiana: Augmented Reality Based In-Situ Annotation of Assembly Videos

“…Many video annotation tools have been developed and used since 1990s, for instance in creating instructional videos [4], [29], analyzing video experiments [30], sharing comments on videos through social networks [31], and generating labeled data for training machine learning models [14]. These tools are typically used for annotation after the video has been captured.…”

“…From the interaction perspective, the mouse and keyboard were the most commonly used input devices before tablets and smartphones with touchscreen become popular. In case of annotating pre-recorded video, it is preferred to use a mouse and a keyboard to achieve precise pointing and fast typing, although users have reported that they become fatigued when using these devices to annotate lengthy video [14]. When users need to use their hands to conduct other tasks simultaneously, hands-free interaction with the annotation tool becomes an essential requirement.…”

“…Instead of designing a novel interaction model to improve user experience, several video annotation tools [14], [35], [36] have focused on developing intelligent features that can reduce manual effort and speed up the annotation process. For example, the video annotation tool presented in [35] used automatic object segmentation to save 78% of manual effort, although the results of the object segmentation algorithms still needed to be manually corrected occasionally.…”

“…tools, materials, machinery components) and instruction for each work step. Compared with Ajalon [14], a state-ofthe-art video annotation tool that supports semi-automatic work step segmentation and object association list extraction, ARiana guided participants to annotate workflow information at least 60% faster without inflating the level of experienced cognitive load in the annotation process. Moreover, the quality of object annotation is significantly higher with ARiana than with Ajalon.…”

“…Section 5 describes the design and development of two main components of ARiana, including context awareness and multi-modal interaction. The system evaluation including a comparison with Ajalon [14] is presented in Section 6. We discuss the implications of the design goals and the findings from the evaluation in Section 7 before we conclude the work in Section 8.…”