Vision-Based Technique and Issues for Multimodal Interaction in Augmented Reality

Abstract

Although many progresses have been accomplished in multimodal interaction, most researchers still treat each modality such as vision and speech, separately. They integrate the results at the application stage. This is because the roles of multiple modalities and their interactions continue to be quantified and precisely understood. However, there are many remaining issues in combining each modality individually. This paper will highlight the main vision problems based on our review for multimodal applications. This review paper will give an overview of the Augmented Reality (AR) technologies which are contributing in most of recent multimodal applications. We cluster vision techniques according to the natural human senses such as face, gesture, and speech that are frequently used in multimodal applications. The main contribution of this paper is to consolidate some of the main issues and approaches in vision-based technique, and to study some of the applications in AR that have been developed within the context of multimodal interaction. We conclude this paper with the future directions.

An Inertial Device-based User Interaction with Occlusion-free Object Handling in a Handheld Augmented Reality

“…Therefore, users can naturally handle the virtual object like in the real world. As agreed by [14], the accurate tracking technique will enhance the user interaction in AR. However, the remaining issues in 3D object manipulation are lighting, occlusion problems and degree of freedom.…”

“…This definition has been extended by Azuma [13] where AR technology should include not only visual input but also the other sensory inputs such as sound, taste, touch or smell. AR is known as an emerging technology to provide a potential solution for natural interaction that allow users to seamlessly interact with virtual contents that been overlaid with our real world in a real-time [14]. AR has been defined as a system that: 1) combines the real and virtual world; 2) is interactive in real time; and 3) is registered in 3D [15] [16].…”

A Review of Multimodal Interaction Technique in Augmented Reality Environment

“…The use of multimodal interaction for AR environment has recently become widespread as can been seen in the works. For instance, three main interaction modalities focused by [13] included gesture recognition, facial recognition and speech recognition. Gesture recognition allowed user to issue commands which may be as simple as a 3D pointer or as complex as the virtual copy of the hand itself.…”

“…To summarise the usage of the types of AR system by the researcher, see-through based system were using HMD [17], [23], [25] and Google glass [13] to visualize the virtual object. Mobile-based system are used by [13], [17], [20]- [22], [27] and tablet [16], [19], [24] as a visualization device. For projection-based system, [26] used table top to display the virtual object and PC-based was used in [14].…”

“…The most popular domains that have been explored are education [13], [15], [25], architecture [13], [15] and games [16], [22], [26]. Furthermore, application domain was averagely has been studied in MMI AR were entertainment, medical, art [13], [15], business [15], [17] and tourism [21], [24]. Meanwhile, maintenance [27] and robotic [23] are the least domain that have applied MMI in AR.…”