Perceived Synchronization of Mulsemedia Services

Abstract

Abstract-Multimediasynchronization involves a temporal relationship between audio and visual media components. The presentation of "in-sync" data streams is essential to achieve a natural impression, as "out-of-sync" effects are often associated with user quality of experience (QoE) decrease. Recently, multi-sensory media (mulsemedia) has been demonstrated to provide a highly immersive experience for its users. Unlike traditional multimedia, mulsemedia consists of other media types (i.e., haptic, olfaction, taste, etc.) in addition to audio and visual content. Therefore, the goal of achieving high quality mulsemedia transmission is to present no or little synchronization errors between the multiple media components. In order to achieve this ideal synchronization, there is a need for comprehensive knowledge of the synchronization requirements at the user interface. This paper presents the results of a subjective study carried out to explore the temporal boundaries within which haptic and air-flow media objects can be successfully synchronized with video media. Results show that skews between sensorial media and multimedia might still give the effect that the mulsemedia sequence is "in-sync" and provide certain constraints under which synchronization errors might be tolerated. The outcomes of the paper are used to provide recommendations for mulsemedia service providers in order for their services to be associated with acceptable user experience levels, e.g. haptic media could be presented with a delay of up to 1 s behind video content, while air-flow media could be released either 5 s ahead of or 3 s behind video content.

Closing the Gap: Human Factors in Cross-Device Media Synchronization

“…Montagud et al extensively reviewed 19 emerging media applications that require inter-destination synchronization from the level of "very high" (10 µs-10 ms) to "low" (500 ms-2000 ms) [22]. In a recent study on perceived synchronization of multi-sensory media, Yuan et al concluded that users may tolerate haptic and air-flow media being one to three seconds behind corresponding video content [39]. A game-with-a-purpose (GWAP) approach was also taken to measure the lower asynchronism (non-synchronicity) as 400 ms for a social TV scenario [26].…”

Video Multimethod Assessment Fusion (VMAF) on 360VR Contents

“…So, many VR applications have been restricted to operate with local video information, although synthesized video could be generated online from delivered abstract representations. Moreover, the synchronized presentations of all multimedia data streams should not be forgotten [4].…”

360° Mulsemedia Experience over Next Generation Wireless Networks - A Reinforcement Learning Approach

“…Studies on QoE performance when considering additional sensory objects to classical video services are provided in [17]. The synchronization between multiple media components such as 2D video, haptic and airflow is analyzed in [18] by measuring the Mean Opinion Score (MOS) for the perceived sense of relevance, reality, distraction, annoyance and enjoyment. In [19], the user perceived QoE denotes that the mulsemedia objects can only partially mask the decrease in movie quality.…”