Making End User Development More Natural

Myers, Brad A.; Ko, Amy J.; Scaffidi, Christopher; Oney, Stephen; Chang, Kerry Shih-Ping; Kery, Mary Beth; Li, Toby Jia-Jun

doi:10.1007/978-3-319-60291-2_1

Cited by 18 publications

(9 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An instructable agent is a promising new type of frame-based agent that can learn intents for new tasks interactively from the end user's natural language instructions [4,37,61] and/or demonstrations [1,35,39,47]. It allows users to use agents for personalized tasks and tasks in "long-tail" domains, addressing the "out-of-domain" errors in human-agent conversations [37].…”

Section: Instructable Agentsmentioning

confidence: 99%

Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs

Chen

Xia

et al. 2020

Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

Self Cite

View full text Add to dashboard Cite

A major problem in task-oriented conversational agents is the lack of support for the repair of conversational breakdowns. Prior studies have shown that current repair strategies for these kinds of errors are often ineffective due to: (1) the lack of transparency about the state of the system's understanding of the user's utterance; and (2) the system's limited capabilities to understand the user's verbal attempts to repair natural language understanding errors. This paper introduces SOVITE, a new multi-modal (speech plus direct manipulation) interface that helps users discover, identify the causes of, and recover from conversational breakdowns using the resources of existing mobile app GUIs for grounding. SOVITE displays the system's understanding of user intents using GUI screenshots, allows users to refer to third-party apps and their GUI screens in conversations as inputs for intent disambiguation, and enables users to repair breakdowns using direct manipulation on these screenshots. The results from a remote user study with 10 users using SOVITE in 7 scenarios suggested that SOVITE's approach is usable and effective.

show abstract

Section: Instructable Agentsmentioning

confidence: 99%

Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs

Chen

Xia

et al. 2020

Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

Self Cite

View full text Add to dashboard Cite

show abstract

“…End-user development of multimodal interaction has become a promising option for end users to develop the support for their own desired tasks. Multimodality is often used to provide "naturalness" in the development process [24] to make it closer to the way the users think about the tasks [11]. Sugilite allows end-users to create voice-activated task automation by demonstrating the task via directly manipulating existing app GUIs [18].…”

Section: Tool Support For Authoring Multimodal Interactionmentioning

confidence: 99%

Geno

Sarmah

Ding

Wang

et al. 2020

Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

Self Cite

View full text Add to dashboard Cite

Supporting voice commands in applications presents significant benefits to users. However, adding such support to existing GUI-based web apps is effort-consuming with a high learning barrier, as shown in our formative study, due to the lack of unified support for creating multimodal interfaces. We present Geno-a developer tool for adding the voice input modality to existing web apps without requiring significant NLP expertise. Geno provides a high-level workflow for developers to specify functionalities to be supported by voice (intents), create language models for detecting intents and the relevant information (parameters) from user utterances, and fulfill the intents by either programmatically invoking the corresponding functions or replaying GUI actions on the web app. Geno further supports multimodal references to GUI context in voice commands (e.g., "move this [event] to next week" while pointing at an event with the cursor). In a study, developers with little NLP expertise were able to add multimodal voice command support for two existing web apps using Geno.

show abstract

“…Interactive task learning (ITL) is an emerging research topic that focuses on enabling task automation agents to learn new tasks and their corresponding relevant concepts through natural interaction with human users (Laird et al, 2017). This topic is also known as end user development (EUD) for task automation (Ko et al, 2011;Myers et al, 2017). Work in this domain includes both physical agents (e.g., robots) that learn tasks that might involve sensing and manipulating objects in the real world (Chai et al, 2018;Argall et al, 2009), as well as software agents that learn how to perform tasks through software interfaces (Azaria et al, 2016;Allen et al, 2007;Leshed et al, 2008).…”

Section: Introductionmentioning

confidence: 99%

Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations

Mitchell

Myers

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations

Self Cite

View full text Add to dashboard Cite

We show SUGILITE, an intelligent task automation agent that can learn new tasks and relevant associated concepts interactively from the user's natural language instructions and demonstrations, using the graphical user interfaces (GUIs) of third-party mobile apps. This system provides several interesting features:(1) it allows users to teach new task procedures and concepts through verbal instructions together with demonstration of the steps of a script using GUIs; (2) it supports users in clarifying their intents for demonstrated actions using GUI-grounded verbal instructions; (3) it infers parameters of tasks and their possible values in utterances using the hierarchical structures of the underlying app GUIs; and (4) it generalizes taught concepts to different contexts and task domains. We describe the architecture of the SUGILITE system, explain the design and implementation of its key features, and show a prototype in the form of a conversational assistant on Android.

show abstract

Making End User Development More Natural

Cited by 18 publications

References 34 publications

Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs

Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs

Geno

Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations

Contact Info

Product

Resources

About