A Wizard of Oz Study Simulating API Usage Dialogues With a Virtual Assistant

Abstract

Virtual Assistant technology is rapidly proliferating to improve productivity in a variety of tasks. While several virtual assistants for everyday tasks are well-known (e.g., Siri, Cortana, Alexa), assistants for specialty tasks such as software engineering are rarer. One key reason software engineering assistants are rare is that very few experimental datasets are available and suitable for training the AI that is the bedrock of current virtual assistants. In this paper, we present a set of Wizard of Oz experiments that we designed to build a dataset for creating a virtual assistant. Our target is a hypothetical virtual assistant for helping programmers use APIs. In our experiments, we recruited 30 professional programmers to complete programming tasks using two APIs. The programmers interacted with a simulated virtual assistant for help -the programmers were not aware that the assistant was actually operated by human experts. We then annotated the dialogue acts in the corpus along four dimensions: illocutionary intent, API information type(s), backward-facing function, and traceability to specific API components. We observed a diverse range of interactions that will facilitate the development of dialogue strategies for virtual assistants for API usage.

Dialogue Management for Interactive API Search

“…An example within Software Engineering research is Wood et al [14], which identified 26 dialogue act types for a hypothetical system to help programmers debug. This paper focuses on API search, and is based on the action space identified by Eberhart et al [15]. That paper presented a Wizard-of-Oz (WoZ) study in which programmers interacted with a hypothetical dialogue system for API usability.…”

“…These actions enable users to search for API components by providing unstructured natural language queries, as well as specific keywords. Duala-Ekoko and Robillard [45], Sadowski et al [46], and Eberhart et al [15] have demonstrated that natural language queries play a more significant role in the API information-seeking process than e.g., signature-matching and faceted search. Furthermore, a broad range of techniques have been investigated supporting semantic search for APIs and source code [47], [48].…”

“…The goals are subdivided into constraints (i.e., what information the user can and cannot provide) and requests (the user's information need); over the course of a dialogue, the constraints and requests are updated as the user collects new information. The agenda is typically structured as a stack; however, we make modifications for our particular search task in order to better capture the API browsing process described by Kelleher and Inchinco [57] and Eberhart et al [15].…”

“…The user simulator processes incoming system dialogue acts by first processing new candidates or evidence, and then identifying the list of available actions by considering whether the task has been completed and whether there are candidate functions to inquire about. Finally, the simulator selects an action from the list using a bigram probability model (i.e., one that returns the probability of a user dialogue act type in response to the most recent system dialogue act type) derived from the Wizard of Oz data collected by Eberhart et al [15].…”

A Neural Question Answering System for Basic Questions about Subroutines

“…A "basic" question is a question about a small detail of a method, such as "what are the parameters to the method convertWavToMp3?" We use basic question because: 1) Eberhart et al [7] found that programmers ask these questions about Java methods during actual programming tasks and isolated five types of these questions, and 2) the "basic" questions provide an excellent stepping-stone problem towards larger problems later. We built question and answer templates and paraphrases based on Eberhart et al's basic question types to construct a dataset for 1.56m Java methods.…”