Prototyping using a Pattern Technique and a Context-Based Bayesian Network in Multimodal Systems

Abstract

Abstract-Today, technology allows us to produce extensive multimodal systems which are totally under human control. These systems are equipped with multimodal interfaces, which enable more natural and more efficient interaction between man and machine. End users can take advantage of natural modalities (e.g. audio, eye gaze, speech, gestures, etc.) to communicate or exchange information with applications. In this work, we assume that a number of these modalities are available to the user. In this paper, we present a prototype of a multimodal architecture, and show how modality selection and fission algorithms are implemented in such a system. We use a pattern technique to divide a complex command into elementary subtasks and select suitable modalities for each of them. We integrate a context-based method using a Bayesian network to resolve ambiguous or uncertain situations.

Context-Based Method Using Bayesian Network in Multimodal Fission System

“…(This case will be illustrated with reel example in section 6.2) Ref. 25 As we mentioned in the previous section, during the fission process, the system should make a decision between several equal choices. In uncertain or ambiguous cases, the BN is an effective solution to solve this problem.…”