Optimization in multimodal interpretation

Abstract

In a multimodal conversation, the way users communicate with a system depends on the available interaction channels and the situated context (e.g., conversation focus, visual feedback). These dependencies form a rich set of constraints from various perspectives such as temporal alignments between different modalities, coherence of conversation, and the domain semantics. There is strong evidence that competition and ranking of these constraints is important to achieve an optimal interpretation. Thus, we have developed an optimization approach for multimodal interpretation, particularly for interpreting multimodal references. A preliminary evaluation indicates the effectiveness of this approach, especially for complex user inputs that involve multiple referring expressions in a speech utterance and multiple gestures.

Cognitive Principles in Robust Multimodal Interpretation

“…Note that, if the status of an object is either Focus or Visible, then T emp(o, e) = 1. This definition of temporal compatibility is different from the function used in our previous work (Chai et al, 2004) that takes real time stamps into consideration. Section 6.2 shows different performance results based on different temporal compatibility functions.…”

“…The term Compatibility(o, e) measures the compatibility between an object o and a referring expression e. Similar to the compatibility measurement in our earlier work (Chai et al, 2004), it is defined by a multiplication of many factors in the following equation:…”

“…The overall compatibility Q(G c , G s ) depends on the node compatibility NodeSim and the edge compatibility EdgeSim, which were further defined by temporal and semantic constraints (Chai et al, 2004). When the algorithm converges, P (α x , α m ) gives the matching probabilities between a referent node α x and a referring node α m that maximizes the overall compatibility function.…”

“…Furthermore, the decision list used in Kehler (2000) is proposed based on data analysis and has not been implemented or evaluated in a real-time system. Therefore, our second goal is to design and implement an efficient algorithm by incorporating these cognitive principles and empirically compare its performance with the optimization approach (Chai et al, 2004), the finite state approach (Johnston & Bangalore, 2000), and the decision list approach (Kehler, 2000).…”

“…In our previous work (Chai, Hong, & Zhou, 2004b;Chai, Hong, Zhou, & Prasov, 2004), a multimodal conversational system was developed for users to acquire real estate information 1 . Figure 1 is the snapshot of a graphical user interface.…”