Poster: Prototyping natural interactions in virtual studio environments by demonstrationby - Combining spatial mapping with gesture following

Abstract

In this work, a rapid prototyping approach for natural real-time interactions in virtual studio environments was used, in order to facilitate the creation of an interactive virtual robotic arm, that could be controlled by hand movements in a natural manner. The approach focuses on tracking a person and manipulating control parameters as a way of steering interactions. It has two main aspects. The first one is the authoring of spatial relationships between the person and the virtual environment. The second aspect is the use of gesture following to synchronize various animations or event sequences inside the virtual environment to the corresponding person's movements. Both aspects allow for a programming-by-demonstration approach and, combined, enable developers to rapidly create interactions by providing examples directly inside the blue or green box of the virtual studio.

Facilitating the creation of natural interactions for live audiovisual performances

“…Although the method was quite powerful, there was, to our knowledge, no application of it in an artistic or musical context. The method seemed to compliment quite well the authoring-bydemonstration strategy presented in [9]. In order to fill that gap, we developed a stand-alone program implementing the method, called OscActivator.…”

“…The use of gesture following in an authoring-by-demontration context was also presented in [9]. A stand-alone tool named OscMotivator was developed as part of that work, which implemented Balivacqua's algorithm, adding more flexibility and features, like for example enabling backwards following and measuring the smoothness of the following as a complimentary detection metric for the gesture.…”

The conductor's philosophy

“…We developed custom gesture recognition software based on the algorithm of Bevilacqua [2]. This allows detecting dynamic gestures that have been previously learned using a simple "specification by demonstration" approach [3]. These gestures are used as input for the algorithmic music composition and control the play of the algorithmic module.…”

Research on Construction of Kinect-Based Virtual Situation Laboratory

“…Currently, a lot of professional virtual situational teaching systems have been developed, such as virtual laboratory dedicated to television studio and Business English Training Room [2,3]. These systems are generally dedicated systems integrating blue-screen image matting, high-definition camera and situation fusion technology in one.…”