Multimodal Cue Detection Engine for Orchestrated Entertainment

Abstract

Abstract. In this paper, we describe a low delay real-time multimodal cue detection engine for a living room environment. The system is designed to be used in open, unconstrained environments to allow multiple people to enter, interact and leave the observable world with no constraints. It comprises detection and tracking of up to 4 faces, estimation of head poses and visual focus of attention, detection and localisation of verbal and paralinguistic events, their association and fusion. The system is designed as a flexible component to be used in conjunction with an orchestrated video conferencing system to improve the overall experience of interaction between spatially separated families and friends. Reduced latency levels achieved to date have shown improved responsiveness of the system.

Models and operators for extension of active multimedia documents via annotations

“…As examples of content-based anno-tations, the processing of audio utterances have been used to perform speech segmentation, word spotting, skimming and topic segmentation [Bouamrane and Luz, 2006] whereas speech recognition has been applied to index discourse using text-based techniques [Kaptein and Marx, 2010]. Video-based sources have been used to index non-verbal interaction, for instance via computer vision to detect behaviors from patterns of eye gaze, gestures and focus of attention [Korchagin et al, 2012]. The use of context-based sources has the purpose of associating external attributes in order to enrich the description or the content of the media elements.…”