Formal description of multi-touch interactions

Abstract

The widespread use of multi-touch devices and the large amount of research that has been carried out around them has made this technology mature in a very short amount of time. This makes it possible to consider multi-touch interactions in the context of safety critical systems. Indeed, beyond this technical aspect, multi-touch interactions present significant benefits such as input-output integration, reduction of physical space, sophisticated multi-modal interaction … However, interactive cockpits belonging to the class of safety critical systems, development processes and methods used in the mass market industry are not suitable as they usually focus on usability and user experience factors upstaging dependability. This paper presents a tool-supported model-based approach suitable for the development of interactive systems featuring multitouch interactions techniques. We demonstrate the possibility to describe touch interaction techniques in a complete and unambiguous way and that the formal description technique is amenable to verification. The capabilities of the notation is demonstrated over two different interaction techniques (namely Pitch and Tap and Hold) together with a software architecture explaining how these interaction techniques can be embedded in an interactive application.

PeNTa: Formal Modeling for Multi-touch Systems Using Petri Net

“…Nam et al showed how to use Coloured Petri Nets (CPN) to achieve hand (data glove) gesture modeling and recognition [14], using Hidden Markov Models (HMM) to recognize gesture features that are then fed to a Petri Net. Petri Nets have been shown to be applicable in event-driven system [15], which is another reason we are interested in them to model input devices. While our approach is based on high-level Petri Nets, Spano et al [16,17] showed how to use Non-Autonomous Petri Nets [18], low-level Petri Nets, for multi-touch interaction.…”

“…While our approach is based on high-level Petri Nets, Spano et al [16,17] showed how to use Non-Autonomous Petri Nets [18], low-level Petri Nets, for multi-touch interaction. Also, Hamon et al [15] expanded on Spano's work, providing more detail to the implementation of Petri Nets for multi-touch interactions.…”

3D Navigation with Six Degrees-of-Freedom using a Multi-Touch Display

“…Coloured Petri Nets (CPN) to achieve hand (data glove) gesture modeling and recognition [151], using HMM to recognize gesture features that are then fed to a Petri Net (PN). PNs have been shown to be applicable in event-driven systems [75], which is another reason they are interesting to use for modeling modern input devices. Spano et al [198,199] showed how to use Non-Autonomous PNs [35] (low-level PNs), for multi-touch interaction.…”

“…Spano et al [198,199] showed how to use Non-Autonomous PNs [35] (low-level PNs), for multi-touch interaction. Also, Hamon et al [75] expanded on Spano's work, providing more detail to the implementation of PNs for multi-touch interactions.…”

“…These reasons might be better understood when PeNTa is placed in the context of other existing approaches for input modeling: Proton and Proton++ [112], Gesture Coder [130], and GestIT [75,197,199]. While PeNTa offers more expressiveness and distributed capabilities, simultaneously providing a solid mathematical framework, the work offered by Proton++, Gesture Coder, and GestIT offers great insight in modeling multi-touch interaction, with different benefits that must be evaluated by the developer.…”