Intelligence methods of multi-modal information fusion in human-computer interaction

Yang, Minghao; Tao, Jianhua

doi:10.1360/n112017-00211

Cited by 9 publications

(8 citation statements)

References 33 publications

(38 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the multimodal fusion study, the fusion methods are mainly divided into the decision layer, the data and the model layer [1]. The research decision layer fusion, Vu et al [32] proposed two single-modality recognitions of speech and gestures through weighting criteria and optimal probability fusion method.…”

Section: B Multimodal Interactionmentioning

confidence: 99%

“…For the system, it is 9 kinds of tasks to understand the intentions of users. We establish the behaviors set as follows: [1,9] n  , the n indicates 9 kinds of user's intentions.…”

Section: A Multimodal Navigational Interaction Overall Frameworkmentioning

confidence: 99%

“…Multimodal fusion interaction is the key to understanding user intention in multi-channel human-computer interaction. Effective fusion is performed according to different modal signals and the user's intention is accurately understood [1]. Therefore, we first fuse multi-modal information, then analyze intentional understanding of the multi-modal, and finally perform voice navigation broadcast interaction based on the intentional understanding.…”

Section: Multimodal Fusion Navigational Interactionmentioning

confidence: 99%

“…The traditional single-channel interaction mode has some shortages, such as poor immersion, inaccurate input information and low efficiency. Alternatively, multichannel interactions can offset the shortcomings of single-channel interactions [1]. For example, many studies use the relationship of information complementation and cross between multiple channels (adding speech to assist gestures for experimental operations).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A Structure Design of Virtual and Real Fusion Intelligent Equipment and Multimodal Navigational Interaction Algorithm

et al. 2020

View full text Add to dashboard Cite

Virtual experiments have become an interesting research topic in the field of education. However, we found that there are some limitations in the current virtual experiments: first, the researchers used the virtual effects of the simulation to represent the virtual experiments, which led to decrease the immersion of the user's simulated experiments; second, most of the virtual experiments are only mouse or touch screen interactive mode, which reduces the realism of user simulation experiments; third, students independently explore the experimental operation process and spend too much time simulating the experiment, which leads to problems such as overloading the operation and low interaction efficiency. In order to solve the above problems, we propose and implement a multimodal navigational interaction virtual and real fusion chemistry laboratory (MNIVRFCL). We design a new sensing structure intelligent equipment and propose a multimodal navigational interaction algorithm (MMNI) based on auditory and tactile channel, which are verified and applied in MNIVRFCL. The MMNI algorithm can detect user's specific behaviors to understand their behavioral intentions, and then the system guide and rectify users' current correct or incorrect behaviors in the form of voice navigation broadcasts. Finally, we achieve the purpose that students can use virtual and real fusion interactions through tactile and auditory channels, they can independently complete simulations and learning experiments based on experimental navigation. The experimental statistic results show that system's successful understanding of intention is 91.48%, and prove the MNIVRFCL operational load reduce by 23.81% compared to the pure virtual experiment, it can reduce the time consumption and improves the students' interaction efficiency.

show abstract

Section: B Multimodal Interactionmentioning

confidence: 99%

“…For the system, it is 9 kinds of tasks to understand the intentions of users. We establish the behaviors set as follows: [1,9] n  , the n indicates 9 kinds of user's intentions.…”

Section: A Multimodal Navigational Interaction Overall Frameworkmentioning

confidence: 99%

Section: Multimodal Fusion Navigational Interactionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Structure Design of Virtual and Real Fusion Intelligent Equipment and Multimodal Navigational Interaction Algorithm

et al. 2020

View full text Add to dashboard Cite

show abstract

“…The multimodal fusion strategy [27] is mainly divided into feature layer fusion and decision layer fusion. In terms of feature layer fusion, in 2010, Jiang et al [28] proposed a multimodal biometric recognition method based on the Laplacian subspace for low-order fusion of face and speech.…”

Section: Related Workmentioning

confidence: 99%

Research on Multimodal Perceptual Navigational Virtual and Real Fusion Intelligent Experiment Equipment and Algorithm

et al. 2020

View full text Add to dashboard Cite

Virtual experiment is an important field of human-computer interaction. With more and more virtual laboratories emerging, we found that problems regarding virtual experiments are rising. Such problems can be listed as follows: First, human-computer interaction has lower efficiency during the process of virtual experiment, which means the computer cannot understand the user's intention thus leading to incorrect operation. Second, there are less detections for false behavior during experiments. Third, the virtual laboratory's sense of operation and realism is not strong. In order to solve the above problems, the multimodal sensing navigation virtual and real fusion laboratory (MSNVRFL) was designed and implemented in this paper. We design a new set of experimental equipment with the function of cognition and study a multimodal fusion model and algorithm for chemical experiments, which are both finally verified and applied in MSNVRFL. By using multimodal fusion perception algorithm, the user's true intentions can be understood and the human-computer interaction efficiency can be improved. By carrying out a virtual experiment with the mold of virtual and real fusion, problems like resources wasting and dangers happened during experiment can be avoided, user's sense of operation and realism can be improved. In addition, teaching navigation and wrong operation behavior reminders are provided for users. The experimental result shows that our method can improve the efficiency of human-computer interaction, reduce the user's cognitive load, strengthen the user's sense of reality and operation and stimulate students' interest in learning. INDEX TERMS Multimodal fusion, virtual experiments, intelligent teaching, human-computer interaction.

show abstract

How Efficiency and Naturalness Change in Multimodal Interaction in Mobile Navigation Apps

Ling

Peng

et al. 2020

Advances in Intelligent Systems and Computing

View full text Add to dashboard Cite

Intelligence methods of multi-modal information fusion in human-computer interaction

Cited by 9 publications

References 33 publications

A Structure Design of Virtual and Real Fusion Intelligent Equipment and Multimodal Navigational Interaction Algorithm

A Structure Design of Virtual and Real Fusion Intelligent Equipment and Multimodal Navigational Interaction Algorithm

Research on Multimodal Perceptual Navigational Virtual and Real Fusion Intelligent Experiment Equipment and Algorithm

How Efficiency and Naturalness Change in Multimodal Interaction in Mobile Navigation Apps

Contact Info

Product

Resources

About