Figurines, a multimodal framework for tangible storytelling

Abstract

This paper presents Figurines, an offline framework for narrative creation with tangible objects, designed to record storytelling sessions with children, teenagers or adults. This framework uses tangible diegetic objects to record a free narrative from up to two storytellers and construct a fully annotated representation of the story. This representation is composed of the 3D position and orientation of the figurines, the position of decor elements and interpretation of the storytellers' actions (facial expression, gestures and voice). While maintaining the playful dimension of the storytelling session, the system must tackle the challenge of recovering the free-form motion of the figurines and the storytellers in uncontrolled environments. To do so, we record the storytelling session using a hybrid setup with two RGB-D sensors and figurines augmented with IMU sensors. The first RGB-D sensor completes IMU information in order to identify figurines and tracks them as well as decor elements. It also tracks the storytellers jointly with the second RGB-D sensor. The framework has been used to record preliminary experiments to validate interest of our approach. These experiments evaluate figurine following and combination of motion and storyteller's voice, gesture and facial expressions. In a make-believe game, this story representation was re-targeted on virtual characters to produce an animated version of the story. The final goal of the Figurines framework is to enhance our understanding of the creative processes at work during immersive storytelling.

Multimodal Observation and Classification of People Engaged in Problem Solving: Application to Chess Players

“…Figure 2 presents the recording setup for our experiment. This system drew on lessons learned from an earlier system used to record children during storytelling sessions [18]. As can be seen from Figure 2, it is composed of several hardware elements: a 23.8 inches Touch-Screen computer, a Kinect 2.0 mounted 35 cm above the screen focusing on the chess player, a 1080p Webcam for a frontal view, a Tobii Eye-Tracking bar (Pro X2-60 screen-based) and two adjustable USB-LED for lighting condition control.…”

Multimodal Observation and Interpretation of Subjects Engaged in Problem Solving

“…Figure 2 presents the recording setup for our experiment. is setup is a derivative version of the one we use to record children during storytelling sessions [17]. As seen, it is composed of several hardware elements: a 23.8" Touch-Screen computer, a Kinect 2.0 mounted 35cm above the screen focusing on the chess player, a 1080p Webcam for a frontal view, a Tobii Eye-Tracking bar (Pro X2-60 screen-based) and two adjustable USB-LED for lighting condition control.…”