Augmenting user interfaces with adaptive speech commands

Abstract

We present a system that augments any unmodified Java application with an adaptive speech interface. The augmented system learns to associate spoken words and utterances with interface actions such as button clicks. Speech learning is constantly active and searches for correlations between what the user says and does. Training the interface is seamlessly integrated with using the interface. As the user performs normal actions, she may optionally verbally describe what she is doing. By using a phoneme recognizer, the interface is able to quickly learn new speech commands. Speech commands are chosen by the user and can be recognized robustly due to accurate phonetic modelling of the user's utterances and the small size of the vocabulary learned for a single application. After only a few examples, speech commands can replace mouse clicks. In effect, selected interface functions migrate from keyboard and mouse to speech. We demonstrate the usefulness of this approach by augmenting jfig, a drawing application, where speech commands save the user from the distraction of having to use a tool palette.

PixelTone

“…These high level commands are manually created and PIXELTONE may be better served by learning from examples [10] or mining online tutorials.…”

“…Since these templates are based on grammatical structures of sentences, it is possible to semi-automate the process of defining templates by analyzing grammatical patterns on large corpora of users' utterances. Alternatively, users can train the system directly by verbalizing what they are doing as they are doing [10].…”

Grounded Semantic Composition for Visual Scenes

“…We are also continuing work in applying our results from grounded language systems to multimodal interface design (Gorniak & Roy, 2003). We recently demonstrated an application of the Bishop system as described in this paper to the problem of referent resolution in the graphical user interface for a 3D modelling application, Blender (Blender Foundation , 2003).…”

Usability evaluation of navigation tasks by people with intellectual disabilities: a Google and SAPO comparative study regarding different interaction modalities

“…Speech-based user interfaces are considered a solid foundation when handling natural interfaces regarding people with some sort of disability: It permits the input of information without the resort to a keyboard or even in the event of the inexistence of a monitor; it facilitates tasks where hands and/or eyes of the users are busy; and it relieves the need for writing for people with motor or intellectual disabilities [34]. Some systems have been developed that take advantage of speech interfaces in order to replace quick commands given by the mouse or keyboard in graphical interfaces, with very positive results [12,20].…”