Introducing semantic information during conceptual modelling of interaction for virtual environments

Abstract

The integration of semantic information in virtual environment interaction is mostly still ad-hoc. The system is usually designed in such a way that the design of the framework incorporates the semantic information which then can be used to utilise these semantics during interaction. We introduce a model-based user interface approach which introduces semantic information, represented using ontologies, during the modelling phase. This semantic information itself is created during the design of the virtual world. The approach we propose is system independent and makes it possible for the semantic information content to be chosen and adapted in complete freedom without considering the underlying framework. We incorporate semantics in NiMMiT, our notation for multimodal interaction modelling. We present two case studies which validate the flexibility of our approach.

An Ontology for Semantic Modelling of Virtual World

“…Vanacken et al [11] introduce the use of semantic information, represented using ontologies, in conceptual modelling of interaction invirtual environments.…”

A serious game model for cultural heritage

“…Approaches from this direction are based on ontologies and semantics [Tutenel et al 2008] in order to give operational meanings to the objects in the VWs. This allows authors to focus better on the specifics of the cultural domain [Vanacken et al 2007]. We have addressed this issue by resorting to a generalization of the task-based learning pedagogical approach, using annotated tasks as reusable game mechanisms suited for focused, situated learning.…”

Lessons from research on interaction with virtual environments

“…Requirements Cavazza (1998) Recognises a need for a knowledge representation layer for virtual environments Pioneering works Thalmann et al (1999) Informed environments based on a database model Urban life simulation Method for building virtual scenes with semantic information NOVAE Allongue, 1997, 1998) Semantic approach to increase VW interoperability Naval simulation (case study) (Soto and Allongue, 2002) Proposes the ontology paradigm Influence/reaction model for describing behaviours and reactions Doyle (2002) Introduces the concept of annotated environment (with structured representations of its contents and purpose) Agent architecture for interacting with the annotated environment Virtual objects with semantics Smart objects (Kallmann and Thalmann, 1999) Pre-programmed possible interactions Extended with action semantics expressed by rules, to reason about the consequences of actions (Abaci et al, 2005) Starfish (Badawi and Donikian, 2004) Synoptic objects Set of actions assigned to interactive surfaces Agents get the data from these interactive surfaces NiMMiT (Vanacken et al, 2007) Semantics incorporated in a diagram based notation intended to describe multimodal interaction Driving simulator (case study) Complete architectures with ontologies Chang et al (2005) Framework with an ontology-based cognitive middle layer between agent minds and the environment manages semantic concepts This layer also represents actions through causal rules Grimaldo et al (2006) Multi-agent framework composed of: ontologies, a semantic layer, planning based agents, and a 3D Engine Grimaldo et al (2008a) Ontology is used to define social relations among agents within an artificial society Virtual university bar (Grimaldo et al, 2008b) (Pellens et al, 2005) Necessity to let the domain expert participate in the specification of the VR application (Troyer et al, 2003) Virtual bowling game (Bille et al, 2004) and virtual shops (Troyer et al, 2007) Approach to design and develop a VR application where the domain expertise is used to generate it (Bille et al, 2004) Development process composed of three sequential steps: the specification step, the mapping step and the generation step (Pellens et al, 2005) method for building virtual scenes with associated semantic information as well as for the exploitation of such scenes. The three-dimensional scene provided by the designer is divided into two parts, one for visualization and another for database construction.…”

“…In Vanacken et al (2007), the authors introduce the usage of semantic information, represented using ontologies, during conceptual modelling of interaction for virtual environments. This semantic information itself is created during the design of the virtual world.…”