MuViSync: Realtime music video alignment

Abstract

In recent years, the popularity of compressed music files and online music downloads has increased dramatically. Today's users own large digital collections of high quality music on their computers and portable devices to be played in their homes or on the go. In addition, music videos are being offered online both for free and through monthly subscriptions, opening up the opportunity to turn the music listening activity into a multimedia experience. The work presented in this paper addresses the challenge of music audio and music video synchronisation. In particular, we have developed a prototype, named MuViSync, to automatically synchronise music videos to the songs that users are listening to in real-time. At the core of the MuViSync prototype are novel audio synchronization algorithms to tackle the differences in tempo, pitch, sampling rates, structure, and introductions and endings that are common in the various digital recordings of the same song in modern music. The music and the music video are initially aligned and then kept in sync within the limits of human perception. In experiments with 320 matching pairs of audio files and music videos, the proposed algorithms successfully synchronise music and video to within 100 milliseconds of each other in over 90% of the cases.

Movie synchronization by audio landmark matching

“…Another state-ofthe-art approach for synchronizing two related contents is Dynamic Time Warping (DTW). As examples, Macrae et al used it for music video alignment [9] and Anguera et al used it for spoken word matching [10]. This technique efficiently tackles the problem of scene addition and removal by finding an optimal path representing the highest similarity between two feature sequences extracted from those contents.…”

Real-time synchronisation of multimedia streams in a mobile device

“…These algorithms are not applicable for our purpose since they are offline in nature. Other modifications to the classical DTW, like [10], [11] and [1] allow for an online alignment. In [10] the application of slope constraints together with a progressive DTW method is used for online synchronisation of two audio files.…”

“…In [1] we showed the feasibility of online synchronisation between audio media and an independent music video by: 1) finding an initial synchronisation between the two media, and 2) making an online alignment between the two to keep them synchronised. In the current paper we propose an alternative algorithm to improve its efficiency and implement the system on a mobile device (in our case, the Apple iPhone 4 and iPad).…”

“…In this paper we solve this problem by proposing an extension of the MuViSync algorithm [1] to run on a mobile phone. In [1] we showed the feasibility of online synchronisation between audio media and an independent music video by: 1) finding an initial synchronisation between the two media, and 2) making an online alignment between the two to keep them synchronised.…”

“…The user can therefore, instead of just listening to his favourite music, watch the corresponding music video at the same time. Indeed, many services nowadays allow for music video streaming, for example Youtube 1 3 ; or music streaming, such as Spotify 4 , Pandora 5 or GrooveShark 6 . Due to bandwidth limitations the audio in the streamed videos is usually of lower quality than when streaming audio alone, and neither is comparable to locally stored high quality audio files or music on audio CDs.…”