Bridging the Gap: Enriching YouTube Videos with Jazz Music Annotations

Abstract

Web services allow permanent access to music from all over the world. Especially in the case of web services with user-supplied content, e.g., YouTube™, the available metadata is often incomplete or erroneous. On the other hand, a vast amount of high-quality and musically relevant metadata has been annotated in research areas such as Music Information Retrieval (MIR). Although they have great potential, these musical annotations are often inaccessible to users outside the academic world. With our contribution, we want to bridge this gap by enriching publicly available multimedia content with musical annotations available in research corpora, while maintaining easy access to the underlying data. Our web-based tools offer researchers and music lovers novel possibilities to interact with and navigate through the content. In this paper, we consider a research corpus called the Weimar Jazz Database (WJD) as an illustrating example scenario. The WJD contains various annotations related to famous jazz solos. First, we establish a link between the WJD annotations and corresponding YouTube videos employing existing retrieval techniques. With these techniques, we were able to identify 988 corresponding YouTube videos for 329 solos out of 456 solos contained in the WJD. We then embed the retrieved videos in a recently developed web-based platform and enrich the videos with solo transcriptions that are part of the WJD. Furthermore, we integrate publicly available data resources from the Semantic Web in order to extend the presented information, for example, with a detailed discography or artists-related information. Our contribution illustrates the potential of modern web-based technologies for the digital humanities, and novel ways for improving access and interaction with digitized multimedia content.

JSD: A Dataset for Structure Analysis in Jazz Music

“…In case of the WJD, many of the recordings are also available on video-sharing platforms such as YouTube. Balke et al (2018) present a web-based approach to identify and link most of the jazz recordings underlying the WJD to corresponding videos on YouTube. 9…”

“…For the purpose of an easy access and a better understanding of the JSD annotations, we created webbased interfaces using similar technologies as the ones described by Balke et al (2018) and Rosenzweig et al (2020). The starting website, as indicated by Figure 1a, provides an overview of the JSD.…”

Cross-Modal Music Retrieval and Applications: An Overview of Key Methodologies

“…For example, structured websites can be used to automatically derive text-, score-, and audio-based queries to look for other musically related documents on the Web [13,30]. Furthermore, YouTube videos may be automatically enriched with manually or automatically generated musical annotations, as recently demonstrated in [31].…”

Understanding Optical Music Recognition

“…• technology which transforms sheet music or printed scores into a machine readable format [1] • automatic recognition and classification of symbolic music notation [2] • system that aims to minimise human involvement in music input. The musical score is scanned to a bitmap image, and the computer attempts to parse the bitmap [5] • form of structured document analysis where symbols overlaid on the conventional five-line stave are isolated and identified so that the music can be played through a MIDI system, or edited in a music publishing system [8] • identifying musical symbols on a scanned sheet of music, and interpreting them so that the music can either be played by the computer, or put into a music editor [9] • system to convert optically scanned pages of music into a versatile machine-readable format [7] • system that aims at converting optically scanned pages of music into a versatile machine-readable format [13] • system that aims at converting the vast repositories of sheet music in the world into an on-line digital format [15] • computer system that can 'read' printed music [10] • system that can be used to convert music scanned from paper into a format suitable for playing or editing on a computer [11] • technique that makes it possible to automatically build indexes on the actual content of sheet music [14] • process to automatically extract symbolic note information from scanned pages [16] • system to convert sheet music images to symbolic music representations [17] • the recognition of music scores [19] • field devoted to transcribe sheet music into some machine-readable format [18] • the process to convert a music score image into a machine-readable format [20] • task of transcribing a music score into a machine readable format [21] • task of recognizing and interpreting printed music and its transformation into MIDI [24] • research directed towards the recognition of printed scores as well as handwritten music notation [23] (Actually referred to as Optical Music Reading) • systems for music score recognition [26] • software that recognises music notation and produces a symbolic representation of music [27] • key problem for coding western music sheets in the digital world [29] • system that aims at saving time in converting hardcopy of the music score into an electronic version [31] • task devoted to convert an image of...…”