An empirical study of speech and gesture interaction

Robbe, Sandrine

doi:10.1145/286498.286815

Cited by 8 publications

(1 citation statement)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While a few studies look at gesture and speech inputs have examined mid-air gestures [2,11,20,27,30,37], some only looked at a subset of gesturing such as pointing gestures [7,52], paddling gestures [21], or two dimensional (2D) gestures [35,52]. The work presented here examines any mid-air gesture and / or utterance that a participant feels is appropriate for a given referent.…”

Section: Gesture and Speech Studiesmentioning

confidence: 99%

Understanding Gesture and Speech Multimodal Interactions for Manipulation Tasks in Augmented Reality Using Unconstrained Elicitation

Williams

Ortega

2020

Proc. ACM Hum.-Comput. Interact.

View full text Add to dashboard Cite

This research establishes a better understanding of the syntax choices in speech interactions and of how speech, gesture, and multimodal gesture and speech interactions are produced by users in unconstrained object manipulation environments using augmented reality. The work presents a multimodal elicitation study conducted with 24 participants. The canonical referents for translation, rotation, and scale were used along with some abstract referents (create, destroy, and select). In this study time windows for gesture and speech multimodal interactions are developed using the start and stop times of gestures and speech as well as the stoke times for gestures. While gestures commonly precede speech by 81 ms we find that the stroke of the gesture is commonly within 10 ms of the start of speech. Indicating that the information content of a gesture and its co-occurring speech are well aligned to each other. Lastly, the trends across the most common proposals for each modality are examined. Showing that the disagreement between proposals is often caused by a variation of hand posture or syntax. Allowing us to present aliasing recommendations to increase the percentage of users' natural interactions captured by future multimodal interactive systems. CCS Concepts: • Human-centered computing → Human computer interaction (HCI); User studies; Mixed / augmented reality; Interaction techniques; Empirical studies in HCI; User centered design.

show abstract