Interaction Proxies for Runtime Repair and Enhancement of Mobile Application Accessibility

Zhang, Xiaoyi; Ross, Anne Spencer; Caspi, Anat; Fogarty, James; Wobbrock, Jacob O.

doi:10.1145/3025453.3025846

Cited by 54 publications

(31 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The overlay is not a common metaphor in interfacesusers are more familiar with static screenshots where nothing is interactive, and actual GUIs where every element can be di rectly interacted with. The overlay (also known as interaction proxy [68]) sits in between, where the user can specify actions about an underlying GUI element (e.g., the value that should go into a textbox or the menu item that should be selected) using direct manipulation, but not directly interact with these GUI elements. A future direction is to explore the design space of overlay interfaces to make them more intuitive to use.…”

Section: Discussionmentioning

confidence: 99%

Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs

Chen

Xia

et al. 2020

Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

View full text Add to dashboard Cite

A major problem in task-oriented conversational agents is the lack of support for the repair of conversational breakdowns. Prior studies have shown that current repair strategies for these kinds of errors are often ineffective due to: (1) the lack of transparency about the state of the system's understanding of the user's utterance; and (2) the system's limited capabilities to understand the user's verbal attempts to repair natural language understanding errors. This paper introduces SOVITE, a new multi-modal (speech plus direct manipulation) interface that helps users discover, identify the causes of, and recover from conversational breakdowns using the resources of existing mobile app GUIs for grounding. SOVITE displays the system's understanding of user intents using GUI screenshots, allows users to refer to third-party apps and their GUI screens in conversations as inputs for intent disambiguation, and enables users to repair breakdowns using direct manipulation on these screenshots. The results from a remote user study with 10 users using SOVITE in 7 scenarios suggested that SOVITE's approach is usable and effective.

show abstract

Section: Discussionmentioning

confidence: 99%

Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs

Chen

Xia

et al. 2020

Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

View full text Add to dashboard Cite

show abstract

“…Once the user finds the target value in an app, they can long press on the target value to select it and indicate it as the target value. PUMICE uses an interaction proxy overlay [53] for recording, so that it can record all values visible on the screen, not limited to the selectable or clickable ones. PUMICE can extract these values from the GUI using the screen scraping mechanism in the underlying SUGILITE framework [20].…”

Section: Support For Concept Learningmentioning

confidence: 99%

PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations

Radensky

Jia

et al. 2019

Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology

108

View full text Add to dashboard Cite

Figure 1. Example structure of how PUMICE learns the concepts and procedures in the command "If it's hot, order a cup of Iced Cappuccino." The numbers indicate the order of utterances. The screenshot on the right shows the conversational interface of PUMICE. In this interactive parsing process, the agent learns how to query the current temperature, how to order any kind of drink from Starbucks, and the generalized concept of "hot" as "a temperature (of something) is greater than another temperature". ABSTRACTNatural language programming is a promising approach to enable end users to instruct new tasks for intelligent agents. However, our formative study found that end users would often use unclear, ambiguous or vague concepts when naturally instructing tasks in natural language, especially when specifying conditionals. Existing systems have limited support for letting the user teach agents new concepts or explaining unclear concepts. In this paper, we describe a new multimodal domain-independent approach that combines natural language programming and programming-by-demonstration to allow users to first naturally describe tasks and associated conditions at a high level, and then collaborate with the agent to recursively resolve any ambiguities or vagueness through conversations and demonstrations. Users can also define new procedures and concepts by demonstrating and referring to contents within GUIs of existing mobile apps. We demonstrate this approach in PUMICE, an end-user programmable agent that implements this approach. A lab study with 10 users showed its usability.

show abstract

“…Prior work on reverse engineering of user interfaces has mainly used sceenshots or screencast videos. These approaches have looked to automatically extract GUI components from screenshot images in order to decouple GUI element representation from predefined image templates [2,9,12,22,38], to augment existing interfaces through understanding of GUI components [2,12], and to extract interaction flows from screencast videos and screen metadata [25,28,29,39]. Prefab [12] identifies GUI elements using GUI-specific visual features, which enables overlaying advanced interaction techniques on top of existing interfaces.…”

Section: Reverse Engineering User Interfacesmentioning

confidence: 99%

“…For instance, FrameWire [29] automatically extracts interaction flows from video recordings of paper prototype user tests. Using Android's accessibility API, Sugilite [28] and Interaction Proxies [39] extract the screen structures, in order to create automation and improve mobile application accessibility. Kim et al [25] apply a crowdsourcing workflow to extract step-by-step structure from existing online tutorial videos.…”

Section: Reverse Engineering User Interfacesmentioning

confidence: 99%

StateLens

Guo

Kong

Rivera

et al. 2019

Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology

View full text Add to dashboard Cite

Blind people frequently encounter inaccessible dynamic touchscreens in their everyday lives that are difficult, frustrating, and often impossible to use independently. Touchscreens are often the only way to control everything from coffee machines and payment terminals, to subway ticket machines and in-flight entertainment systems. Interacting with dynamic touchscreens is difficult non-visually because the visual user interfaces change, interactions often occur over multiple different screens, and it is easy to accidentally trigger interface actions while exploring the screen. To solve these problems, we introduce StateLens -a three-part reverse engineering solution that makes existing dynamic touchscreens accessible. First, StateLens reverse engineers the underlying state diagrams of existing interfaces using point-of-view videos found online or taken by users using a hybrid crowd-computer vision pipeline. Second, using the state diagrams, StateLens automatically generates conversational agents to guide blind users through specifying the tasks that the interface can perform, allowing the StateLens iOS application to provide interactive guidance and feedback so that blind users can access the interface. Finally, a set of 3Dprinted accessories enable blind people to explore capacitive touchscreens without the risk of triggering accidental touches on the interface. Our technical evaluation shows that State-Lens can accurately reconstruct interfaces from stationary, hand-held, and web videos; and, a user study of the complete system demonstrates that StateLens successfully enables blind users to access otherwise inaccessible dynamic touchscreens.

show abstract

Interaction Proxies for Runtime Repair and Enhancement of Mobile Application Accessibility

Cited by 54 publications

References 54 publications

Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs

Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs

PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations

StateLens

Contact Info

Product

Resources

About