An approach integrating two complementary model-based environments for the construction of multimodal interactive applications

Abstract

Abstract. This paper presents a tool suite for the engineering of multimodal Post-WIMP Interactive Systems. The work presented here extends previous work done on design, prototyping, specification and verification of interactive systems and integrates two previously unrelated approaches. The first element of this integration is ICOM (a dataflow model dedicated to low-level input modelling) and its environment ICON which allows for editing and simulating ICOM models. The other element is ICOs (a formal description technique mainly dedicated to dialogue modelling) and its environment PetShop which allows for editing, simulating and verifying ICOs models. This paper shows how these two approaches have been integrated and that this integration allows for engineering multimodal interactive systems. We show on a Range Slider case study how these tools can be used for prototyping interactive systems in general and multimodal interaction techniques in particular. We also present in details how the changes in the interaction techniques impact the models at various levels of the software architecture.

HITPROTO: a tool for the rapid prototyping of haptic interactions for haptic data visualization

“…Navarre et al [24] explain that with the ICon notation alone it is "difficult to deal with the representation of the set of states the system can be in and how events produced by the user through the use of input devices make the states evolves". Hence, they integrated the ICoM model (and ICon environment) with the ICO formal description technique (and the PetShop environment), which through Petri-nets describes the system's states and their changes.…”

Design and usability evaluation of multimodal interaction with finite state machines: a conceptual framework

“…The use of formal techniques to model interaction [9,16,23] allows for describing systems in a complete and nonambiguous way, and opens the possibility of quantitative assessment. According to [23], the multiple facets of multimodal interfaces, and their strong temporal constraints are difficult to capture with classical formal description techniques.…”

Usability Study of Multi-modal Interfaces Using Eye-Tracking

“…The formal specification of the behavior of such multimodal interaction techniques is given in [22].Beyond these software engineering aspects the usability of the sytem was one of the main concern of this project. Indeed, such safety critical systems require certification by independent authorities prior to deployment and human factor certification is a critical aspect of the certification phase.…”