Implementing interface attachments based on surface representations

Abstract

This paper describes an architecture for supporting interface attuchments -small interactive programs which are designed to augment the functionality of other applications. This architecture is designed to work with a diverse set of conventional applications, but require only a minimal set of "hooks" into those applications. In order to achieve this, the work described here concentrates on what we will call observational attachments, a subclass of attachments that operate primarily by observing and manipulating the surface representations of applicationsthat is the visual information that applications would normally display on the screen or print. These attachments can be thought of as "looking over the shoulder of the user" to assist with various tasks. By requiring very little modification to, or help from, the applications they augment, this approach supports the creation of a set of uniform services that can be applied across a more diverse set of applications than traditional approaches.

Domain-Independent Programming by Demonstration in Existing Applications

“…Analogous requirements have been identified for intelligent tutoring systemsobservation, inspection, and scripting (Ritter and Koedinger 1995;Cheikes et al 1998). User studies that record user actions (Kay and Thomas 1995;Linton, Charron, and Joy 1999), animated help systems that require complete and exclusive control of the user interface (Bharat and Sukaviriya 1993;Miura and Tanaka 1998), and attachable, application-independent tools (Olsen et al 1999) make similar demands.…”

“…AppleScript is an application-independent, high-level scripting language for the Apple Macintosh (Apple Computer 1993-1999. Familiar uses it to record the user's actions and to examine and control target applications.…”

Adaptive sliding menubars make existing software more accessible to people with severe motion impairments

“…However, Johar can only be used for applications that are explicitly designed to cater to the Johar interface. Olsen et al [13] described an architecture for creating ''interface attachments''-small independent programs, such as a text searcher or a spell checker, that can augment the functionality of a variety of applications. Their implementation involves intercepting components of a Java user interface toolkit in order to access the visual information that the applications display on screen.…”

“…Menu Controller behaves in a similar way: when a user clicks off of Menu Controller and onto the main window of the application, Menu Controller again renders the root menu entries of that application. Univ Access Inf Soc (2014) 13:5-22 13 The sequence of large buttons displayed by Menu Controller have a sliding functionality. The toolbar has two arrow buttons: a ''Prev'' and a ''Next'' button that enable the user to ''slide'' the toolbar across the screen, that is, collectively moving the positions of the menu buttons on the screen.…”

A general-purpose target-aware pointing enhancement using pixel-level analysis of graphical interfaces

“…The API also exposes widget models, not necessarily their on-screen view (e.g., the pixel coordinates of a slider's thumb are intentionally encapsulated). Techniques for dynamic code modification can also often be made to work in a single application or toolkit [13,14,32], but are generally too brittle for enhancement of the full desktop. Code injection techniques also sometimes fail (e.g., when a skinnable application uses pre-rendered images of widgets instead of meaningful graphics operations).…”