An Evaluation of an Augmented Reality Multimodal Interface Using Speech and Paddle Gestures

Irawati, Sylvia; Green, Scott; Billinghurst, Mark; Duenser, Andreas; Ko, Heedong

doi:10.1007/11941354_28

Cited by 51 publications

(28 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The final system allowed a user to pick and place virtual furniture in an AR scene using a combination of paddle gestures and speech commands. Irawati et al conducted a pilot user study on the benefits of multimodal interaction [17]. However, their system did not support natural free hand input and users had to memorize or refer a list of commands to interact with virtual objects.…”

Section: Related Workmentioning

confidence: 99%

A Wizard of Oz study for an AR multimodal interface

Lee

Billinghurst

2008

Proceedings of the 10th International Conference on Multimodal Interfaces

Self Cite

View full text Add to dashboard Cite

In this paper we describe a Wizard of Oz (WOz) user study of an Augmented Reality (AR) interface that uses multimodal input (MMI) with natural hand interaction and speech commands. Our goal is to use a WOz study to help guide the creation of a multimodal AR interface which is most natural to the user. In this study we used three virtual object arranging tasks with two different display types (a head mounted display, and a desktop monitor) to see how users used multimodal commands, and how different AR display conditions affect those commands. The results provided valuable insights into how people naturally interact in a multimodal AR scene assembly task. For example, we discovered the optimal time frame for fusing speech and gesture commands into a single command. We also found that display type did not produce a significant difference in the type of commands used. Using these results, we present design recommendations for multimodal interaction in AR environments.

show abstract

Section: Related Workmentioning

confidence: 99%

A Wizard of Oz study for an AR multimodal interface

Lee

Billinghurst

2008

Proceedings of the 10th International Conference on Multimodal Interfaces

Self Cite

View full text Add to dashboard Cite

show abstract

“…Irawati et al [13] developed a computer vision based AR system with multimodal input, allowing a user to pick and place virtual furniture in an AR scene using a combination of paddle gestures and speech commands. In the evaluation study they found that multimodal input enabled subjects to complete a task faster than with gesture alone.…”

Section: Multimodal Gesture and Speech Interfaces In Armentioning

confidence: 99%

Grasp-Shell vs gesture-speech: A comparison of direct and indirect natural interaction techniques in augmented reality

Piumsomboon

Altimira

Kim

et al. 2014

2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)

Self Cite

View full text Add to dashboard Cite

In order for natural interaction in Augmented Reality (AR) to become widely adopted, the techniques used need to be shown to support precise interaction, and the gestures used proven to be easy to understand and perform. Recent research has explored free-hand gesture interaction with AR interfaces, but there have been few formal evaluations conducted with such systems.In this paper we introduce and evaluate two natural interaction techniques: the free-hand gesture based Grasp-Shell, which provides direct physical manipulation of virtual content; and the multi-modal Gesture-Speech, which combines speech and gesture for indirect natural interaction. These techniques support object selection, 6 degree of freedom movement, uniform scaling, as well as physics-based interaction such as pushing and flinging.We conducted a study evaluating and comparing Grasp-Shell and Gesture-Speech for fundamental manipulation tasks. The results show that Grasp-Shell outperforms Gesture-Speech in both efficiency and user preference for translation and rotation tasks, while Gesture-Speech is better for uniform scaling. They could be good complementary interaction methods in a physics-enabled AR environment, as this combination potentially provides both control and interactivity in one interface. We conclude by discussing implications and future directions of this research.

show abstract

“…The user wore a data glove, head mounted display and viewpoint tracking equipment. Irawati et al [6] developed a computer vision based AR systems with multimodal input, allowing a user to pick and place virtual furniture in an AR scene using a combination of paddle gestures and speech commands. In the evaluation study they found that multimodal input enabled subjects to complete a task faster than with gesture alone.…”

Section: Related Workmentioning

confidence: 99%

Hands and speech in space

Billinghurst

2013

Proceedings of the 15th ACM on International Conference on Multimodal Interaction

Self Cite

View full text Add to dashboard Cite

Augmented Reality (AR) is technology that allows virtual imagery to be seamlessly integrated into the real world. Although first developed in the 1960's it has only been recently that AR has become widely available, through platforms such as the web and mobile phones. However most AR interfaces have very simple interaction, such as using touch on phone screens or camera tracking from real images. New depth sensing and gesture tracking technologies such as Microsoft Kinect or Leap Motion have made is easier than ever before to track hands in space. Combined with speech recognition and AR tracking and viewing software it is possible to create interfaces that allow users to manipulate 3D graphics in space through a natural combination of speech and gesture. In this paper I will review previous research in multimodal AR interfaces and give an overview of the significant research questions that need to be addressed before speech and gesture interaction can become commonplace.

show abstract

An Evaluation of an Augmented Reality Multimodal Interface Using Speech and Paddle Gestures

Cited by 51 publications

References 25 publications

A Wizard of Oz study for an AR multimodal interface

A Wizard of Oz study for an AR multimodal interface

Grasp-Shell vs gesture-speech: A comparison of direct and indirect natural interaction techniques in augmented reality

Hands and speech in space

Contact Info

Product

Resources

About