A Review of Multimodal Interaction

Abstract

Abstract-Due to the contribution to a more natural and efficient human-computer interaction style, multimodal interactive systems are becoming more and more commonplace and cover a wide range of applicants. Therefore, there seems to be an urgent need for an introduction to related concepts of multimodal interaction and an analytical review of the-state-of-the-art. In this paper, we try to present a brief review of some of the significant aspects in multimodal interaction. At first, the definition and advantages of multimodal systems are discussed, and then a brief development history of multimodal interaction is introduced. Next, we touch on the recent development in the breadth of work on input/output modalities and fusion engine. Finally, we provide a personal outlook for the research of multimodal interaction. It is our hope for this paper to provide a picture of the contemporary state of multimodal interaction research and a meaningful outlook for this research field.

Multimodal Interaction Systems Based on Internet of Things and Augmented Reality: A Systematic Literature Review

“…Thirdly, AR and IoT enable otherwise unperceivable information (e.g., readings from sensors, ratings of a service from other users) to be expressed through perceivable modalities (e.g., visualizations, sounds, vibrations) [16]. Finally, based on our analysis of previous literature reviews on MI systems [5,6,19,20], there is a lack of knowledge in the field on the use of AR and IoT in MI systems, as well as on the development of such systems.…”

“…In order to achieve this, it is necessary to perform a systematic review of previous MI system development efforts, which can help shed light on the optimal design of these systems. There have been several attempts [5,6,19,20] to systematically review general research on MI systems; however, only several studies have reviewed the use of IoT and AR in MI systems [21,22].…”

“…(3) "touch", which indicates pressure and actions performed by fingers; and (4) "motion capture", which records movement of the target (sender) by sensor devices. Liang et al [20] presented a similar list of input modalities by including biosignal sensors, such as the electroencephalogram, electromyogram, and electrooculogram. Hogan and Hornecker [19] conducted a systematic review that focused on output modalities used to engage users with materials in art exhibitions.…”

“…However, we noted different uses of terminology for each modality, as the classification of modalities was dependent on the perspectives of the researchers. For example, Liang et al [20] described touch-based interaction as a "haptic-based" modality, whereas Jaimes and Sebe [5] described the same interaction as a "touch" modality. Therefore, a taxonomy of modalities is necessary, which was developed by Augstein and Neumayr [29], who presented a taxonomy of input and output modalities used in HCI.…”

“…We also used the taxonomy of input and output modalities that was proposed by Augstein and Neumayr [29] to avoid inconsistencies in the terminology used by different researchers. For example, "touch" modality has been referred to as both a "haptic-based" modality [20] and "touch" modality [5].…”