InteSe: An Integrated Model for Resolving Ambiguities in Multimodal Sentences

DAMA: A Dynamic Classification of Multimodal Ambiguities

“…This paper deals with the identification of the meaningful features of multimodal ambiguities classes and their dynamic identification. The role of ambiguity classification process for human-machine interaction has been explained in [10] as the first step of a systemic approach to the solution process of any ambiguous multimodal sentence (i.e., the linguistic unit structuring the different multimodal inputs) [21]. The introduction of a classificatory step before the ambiguity solution allows adopting a systematic and modular approach.…”

“…Since managing ambiguity is a complex process, the identification of a specific type of ambiguities enables supporting the interpretation process by detecting the features to be managed and, therefore, optimizing the solution process [21]. In particular, the management of ambiguities can be divided into two sub problems: the first one deals with the identification of the features to be managed through the ambiguity classification step, and once identified, these features may be appropriately managed through the ambiguity solution step (second sub problem).…”

“…Since the proposed method is based on the representation of each multimodal sentence as a string of symbols of the used language, some concepts and notions need to be introduced. Fundamental is the concept of multimodal sentence [21], which is composed of a set of terminal elements that are the elementary parts of a multimodal language [57]. As defined in [21], each terminal element (E i ) is identified by a set of distinctive attributes that we refer as features.…”

From Modal to Multimodal Ambiguities: a Classification Approach

“…In particular the semantic meaning of the element is given considering a domain ontology that provides the conceptual structure of the context. Therefore, a terminal element E i is defined in [14] For example, let the concept "river" be given and the multimodal input defined both by the sketch and the speech modalities. The representation corresponding to the sketch modality is ( ), while the representation related to the speech modality is the signal connected with the word river ( "river").…”

“…If the Multimodal Interpreter produces more than one interpretation of the user's input, then the Multimodal Ambiguities Classification manages the ambiguous interpretation providing its classification. Then, information about the classified ambiguous input and the set of candidate interpretations are transferred to the Multimodal Ambiguities Solution module; to solve ambiguities it was adopted the module InteSe [14], which selects only one interpretation. The interpreted input will produce a response by the system (System Response Generation module).…”