Crowdsourcing authoring of sensory effects on videos

Abstract

Human perception is inherently multi-sensorial involving five traditional senses: sight, hearing, touch, taste, and smell. In contrast to traditional multimedia, based on audio and visual stimuli, mulsemedia seek to stimulate all the human senses. One way to produce multi-sensorial content is authoring videos with sensory effects. These effects are represented as metadata attached to the video content, which are processed and rendered through physical devices into the user's environment. However, creating sensory effects metadata is not a trivial activity because authors have to identify carefully different details in a scene such as the exact point where each effect starts, finishes, and also its presentation features such as intensity, direction, etc. It is a subjective task that requires accurate human perception and time. In this article, we aim at finding out whether a crowdsourcing approach is suitable for authoring coherent sensory effects associated with video content. Our belief is that the combination of a collective common sense to indicate time intervals of sensory effects with an expert fine-tuning is a viable way to generate sensory effects from the point of view of users. To carry out the experiment, we selected three videos from a public mulsemedia dataset, sent them to the crowd through a cascading microtask approach. The results showed that the crowd can indicate intervals in which users agree that there should be insertions of sensory effects, revealing a way of sharing authoring between the author and the crowd.

Beyond Multimedia Authoring

“…Aiming at finding out whether a crowdsourcing approach can support mulsemedia authoring, particularly sensory effect annotation associated with a specific video, authors in [29] present a web platform, called CrowdMuse. This platform collects sensory effect annotations from the crowd allowing authors and users to work together in the mulsemedia authoring phase.…”

“…Furthermore, we can combine manual specification of authors with a crowdsourcing approach. In [29], users give suggestions about time intervals in which specific sensory effects could be activated according to an audiovisual content provided by mulsemedia application authors. This approach allows authors to enhance their specifications of time intervals in which sensory effects should be rendered with audiovisual contents.…”

A Conceptual Architecture and a Framework for Dealing with Variability in Mulsemedia Systems

“…Taipei, Taiwan, and at ACM Multimedia'19 5 , in Nice, France. He was also a session volunteer for ACM Multimedia'19 6 .…”

Integrating Crowdsourcing and Human Computation for Complex Video Annotation Tasks

“…The first case used four different tasks to produce videos enriched with multimedia features by adding additional content and navigation on raw videos [2]. The second case used two different tasks to analyze videos and produce MPEG-V file with metadata about sensory effects of wind and vibration [3]. The third case used two different tasks to produce, from sign language videos, accessible multimedia presentation with navigation on segmented gestures.…”