Understanding Multimodal User Gesture and Speech Behavior for Object Manipulation in Augmented Reality Using Elicitation

Conceptualizing Interactions of Augmented Reality Solutions

“…D2 Hardware: The implemented hardware significantly impacts how the user interacts with the AR solution. Four major characteristics have been identified: mobile AR [9,11,14,25,26,27,28,29,30,31,32,33,34], such as smartphone or tablets, and HMDs, including monocular [29,30,35] and binocular [11,14,26,28,29,30,31,31,32,33,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52], are the most common hardware options. Projection-based AR [14,24,53] and desktop AR [14,31,…”

“…The most common are voice, touch, and gestures. Voice input [11,25,26,30,38,39,47,49,50,51,55,56,57,58,59,60] can be command-based or natural language processing (NLP) [56]. Touch interaction [39,46,48,51,53,56,57,59,60,61] can include both near and far touch.…”

The Cost of Production in Elicitation Studies and the Legacy Bias-Consensus Trade off

“…Those videos are later analyzed based on the gestures used to generate a dataset. Video annotation has traditionally been done by hand [9,10]; however, some recent work has used skeletal data and computer vision [6]. The annotated gestures are then binned into equivalence classes based on their similarity [1,3].…”

“…That evidence helps to fuel elicitation methodology's increase in popularity. Out of this popularity, elicitation has seen use across a variety of domains including: multi-touch [9], internet of things [21], augmented reality [10,11,19], and interactive rings [22].…”

“…In gesture elicitation studies these will be gesture proposals for multi-touch devices [3,8], or mid-air gestures [9]. Other modalities such as gesture and speech [10][11][12][13] can be elicited. Upon the generation of a proposal, the experimenter will trigger the reaction of the system to that proposal.…”