A mulsemedia framework for delivering sensory effects to heterogeneous systems

Abstract

Technological advances in computing have allowed multimedia systems to create more immersive experiences for users. Beyond the traditional senses of sight and hearing, researchers have observed that the use of smell, taste, and touch in such systems is becoming increasingly well-received, leading to a new category of multimedia systems called mulsemedia-multiple sensorial media-systems. In parallel, these systems introduce heterogeneous technologies to deliver different sensory effects such as lighting, wind, vibration, and smell, under varied conditions and restrictions. This new paradigm shift poses many challenges, mainly related to mulsemedia integration, delay, responsiveness, sensory effects intensities, wearable and other heterogeneous devices for delivering sensory effects, and remote delivery of mulsemedia components. In addition, new approaches to interacting with multimedia applications have emerged such as multi-touch interfaces, voice processing, and brain-computer interfaces, giving rise to new kinds of complex interactive systems. In this article, we underpin fundamental challenges for delivering multisensory effects to heterogeneous systems. We propose an interoperable mulsemedia framework for coping with these challenges, meeting the emerging requirements. It is achieved through the evolution of an open distributed mulsemedia system. We changed its core following architectural and design patterns to accommodate different profiles of communication, connectivity, and sensory effects metadata standard according to the need of mulsemedia applications and devices available in the user's environment. The results include case studies where the framework has been duly applied.

Do I Smell Coffee? The Tale of a 360° Mulsemedia Experience

“…To overcome this hurdle, we have designed an end-user prototype as illustrated in Figure 2 that enriches the 360º video viewing by adding other sensory effects like scent and wind, which enables the users to immerse themselves more deeply into the experience. We took into account the work of Saleme et al 14,15 , who have explored both mulsemedia hardware and software solutions and have found gaps in the literature mainly related to reuse and adaptability of mulsemedia solutions. They proposed a framework named PlaySEM Sensory Effects Renderer (SER) to integrate different applications to varied devices in a straightforward fashion by reusing their mulsemedia renderer that supports MPEG-V, a standard to describe Sensory Effects Metadata (SEM), and provides the necessary protocols to interact with our application.…”

“…PlaySEM SER and the 360º application communicate via WebSocket. After opening the videos on the 360º application, their correspondent SEMs were sent to PlaySEM SER, which in turn, pre-processed the SEMs converting them to commands to handle the specific devices configured for the experiment, as described by Saleme et al 15 . Upon the end of this process, the mulsemedia renderer sent a cue to the 360º mulsemedia application to start playing the video.…”

Sumarização de Dispositivos de Efeitos Multissensoriais para Interações Humano-Computador

“…Solucionado a geração de efeitos, outro problema evidente é como integrar as soluções individuais de cada dispositivo para se criar um ambiente multissensorial. Neste sentido, Saleme et al [8] propõem um framework a fim de endereçar o problema da heterogeneidade dos dispositivos e Saleme et al [7] disponibilizam um tutorial para guiar o desenvolvimento de seu próprio ambiente multissensorial. Além disso, a concepção e coordenação dos efeitos nestes ambientes e a avaliação da efetividade destes efeitos na comunicação são alguns dos desafios a serem vencidos nestes novos ambientes de interação humano-computador.…”

A Conceptual Architecture and a Framework for Dealing with Variability in Mulsemedia Systems

“…The fact is that building such systems imposes several challenges and requirements (discussed in the thesis) due mainly to their ever-evolving and heterogeneous traits [Saleme et al 2019d]. As a result, few mulsemedia systems have remained reusable to be applied to different research purposes as opposed to the use of open mulsemedia datasets [Saleme et al 2019e].…”

“…It provides services to mulsemedia applications through a transparent communication broker and is able to work with timeline-and event-based applications. Moreover, the framework presents a set of configurable parameters to customize the solution according to the needs imposed by mulsemedia applications, offers a mechanism to calibrate delay for each component it works with, and provides a debug and flexible simulation scheme [Saleme et al 2019e]. Then, a performance assessment was carried out on communication protocols for the integration between event-based applications, whereby temporal restrictions play a role, and the framework.…”