Using speech to identify gesture pen strokes in collaborative, multimodal device descriptions

Abstract

One challenge in building collaborative design tools that use speech and sketch input is distinguishing gesture pen strokes from those representing device structure, that is, object strokes. In previous work, we developed a gesture/object classifier that uses features computed from the pen strokes and the speech aligned with them. Experiments indicated that the speech features were the most important for distinguishing gestures, thus indicating the critical importance of the speech–sketch alignment. Consequently, we have developed a new alignment technique that employs a two-step process: the speech is first explicitly segmented (primarily into clauses), then the segments are aligned with the pen strokes. Our speech segmentation step is unique in that it uses sketch features for locating segment boundaries in multimodal dialog. In addition, it uses a single classifier to directly combine word-based, prosodic (pause), and sketch-based features. In the second step, segments are initially aligned with strokes based on temporal correlation, and then classifiers are used to detect and correct two common alignment errors. Our two-step technique has proven to be substantially more accurate at alignment than the existing technique that lacked explicit segmentation. It is more important that, for nearly all cases, our new technique results in greater gesture classification accuracy than the existing technique, and performed nearly as well as the benchmark manual speech–sketch alignment.

Speech analysis for conceptual CAD modeling using multi-modal interfaces: An investigation into Architects’ and Engineers’ speech preferences

“…The system proposed the use of commands such as “Create a blue square there” allowing users to employ vague language and use gestures for disambiguation. Speech has been used alongside gesture pen strokes as demonstrated in Herold and Stahovich's study (2011) in AIEDAM's special issue on the Role of Gesture in Designing. Recent studies using multi-modal input for CAD modeling include the studies of Menegotto (2015) who integrated speech with AutoCAD and Nanjundaswamy et al .…”

Interaction Paradigms Versus Age-Related User Profiles: an Evaluation on Content Selection

“…Recentes interfaces multimodais reconhecem até dois modos de interação em diferentes áreas de interesse. Seja em combinações de pens e fala [9], [10], fala e gestos [11], [12], multitoque e objetos tangíveis [13], [14].…”

Sketching across design domains: Roles and formalities

“…Tang (1989, 1991; Tang & Leifer, 1988), Bly (1988), Minneman (1991), and Neilson and Lee (1994) have observed that designers use speech, sketches, and gestures in combination, using each mode to explain and disambiguate the others. To fully understand a communication act the interaction of different media needs to be analyzed (see, e.g., Herold & Stahovich, 2011). Studies of solitary engineering sketching (Pache, 2001) have seen a wide variety of different sketching behavior and ability, with evidence for the reinterpretation of ambiguous notation in only a small number of cases.…”