Towards Linking the Lakh and IMSLP Datasets

Tsai, TJ

doi:10.1109/icassp40776.2020.9053815

Cited by 11 publications

(5 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this set of experiments, we test our method on incomplete queries, modelling a realistic scenario where one may not have the entire piece for a search, but a fragment of an unknown audio recording or an unlabelled page of sheet music, and wishes to identify its originating piece. A similar study on MIDI-to-score retrieval was recently described in [11].…”

Section: Experiments 3: Fragment-level Retrievalmentioning

confidence: 70%

See 1 more Smart Citation

Exploiting Temporal Dependencies for Cross-Modal Music Piece Identification

Carvalho¹,

Widmer²

2021

Preprint

View full text Add to dashboard Cite

This paper addresses the problem of cross-modal musical piece identification and retrieval: finding the appropriate recording(s) from a database given a sheet music query, and vice versa, working directly with audio and scanned sheet music images. The fundamental approach to this [1] is to learn a crossmodal embedding space with a suitable similarity structure for audio and sheet image snippets, using a deep neural network, and identifying candidate pieces by cross-modal near neighbour search in this space. However, this method is oblivious of temporal aspects of music. In this paper, we introduce two strategies that address this shortcoming. First, we present a strategy that aligns sequences of embeddings learned from sheet music scans and audio snippets. A series of experiments on whole piece and fragment-level retrieval on 24 hours worth of classical piano recordings demonstrates significant improvement. Second, we show that the retrieval can be further improved by introducing an attention mechanism to the embedding learning model that reduces the effects of tempo variations in music. To conclude, we assess the scalability of our method and discuss potential measures to make it suitable for truly large-scale applications.

show abstract

Section: Experiments 3: Fragment-level Retrievalmentioning

confidence: 70%

“…Then we measure the final MRR value and the average search time per query. We repeat this procedure 10 times for each subset size and use the average of the results, similarly to [11].…”

Section: Experiments 4: Scalabilitymentioning

confidence: 99%

Exploiting Temporal Dependencies for Cross-Modal Music Piece Identification

Carvalho¹,

Widmer²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The bootleg score features for audio-sheet retrieval are a slight variant of the features described above (Tsai, 2020). There are two differences between the audio and sheet music (alignment) bootleg score representations that prevent their use in a cross-modal hashing framework.…”

Section: Feature Extractionmentioning

confidence: 99%

“…The second step is to identify the sheet music in IMSLP that matches the audio query. Our approach combines ideas from three recent papers: (1) automatic piano transcription using the Onsets and Frames system (Hawthorne et al, 2018), (2) bootleg score features modified for large-scale MIDI-sheet image retrieval (Tsai, 2020), and (3) dynamic n-gram fingerprinting (Yang and Tsai, 2020), which has recently demonstrated sub-second image-image retrieval of IMSLP piano sheet music. While this part of our system simply adapts and combines existing techniques, we believe this is the first empirical study of audio-sheet image retrieval at the scale of IMSLP and can serve as a useful baseline for the research community.…”

Section: Retrievalmentioning

confidence: 99%

Automatic Generation of Piano Score Following Videos

Shan

Tsai

2021

Transactions of the International Society for Music Information Retrieval

Self Cite

View full text Add to dashboard Cite

This article studies the problem of generating a piano score following video from an audio recording in a fully automated manner. This problem contains two components: identifying the piece and aligning the audio with raw sheet music images. Unlike previous work, we focus primarily on working with raw, unprocessed sheet music from IMSLP, which may contain filler pages, other unrelated pieces or movements, or repeats and jumps whose locations are unknown a priori. To solve this problem, we combine state-ofthe-art methods with a novel alignment algorithm called Hierarchical DTW to handle discontinuities, which are the bottleneck on system performance. Hierarchical DTW handles repeats and jumps by considering all line breaks as possible jump locations, and it applies DTW at both a feature level and a segment level. We evaluate our algorithm with 200 PDFs from IMSLP and real audio recordings from Youtube. Our experiments show that Hierarchical DTW consistently outperforms a previously proposed Jump DTW algorithm in handling various types of discontinuities. We present extensive experimental results and analysis of the proposed algorithm.

show abstract

“…While much of the research involving sheet music images revolves around optical music recognition (OMR) [1], there are a lot of other interesting applications and problems that can be solved without requiring OMR as an initial preprocessing step. Some recent examples include piece identification based on a cell phone picture of a physical page of sheet music [2], audio-sheet image score following and alignment [3,4], and finding matches between the Lakh MIDI dataset and the International Music Score Library Project (IMSLP) database [5]. This article explores a composer style classification task based on raw sheet music images.…”

Section: Introductionmentioning

confidence: 99%

A Deeper Look at Sheet Music Composer Classification Using Self-Supervised Pretraining

Yang

Tsai

2021

Applied Sciences

View full text Add to dashboard Cite

This article studies a composer style classification task based on raw sheet music images. While previous works on composer recognition have relied exclusively on supervised learning, we explore the use of self-supervised pretraining methods that have been recently developed for natural language processing. We first convert sheet music images to sequences of musical words, train a language model on a large set of unlabeled musical “sentences”, initialize a classifier with the pretrained language model weights, and then finetune the classifier on a small set of labeled data. We conduct extensive experiments on International Music Score Library Project (IMSLP) piano data using a range of modern language model architectures. We show that pretraining substantially improves classification performance and that Transformer-based architectures perform best. We also introduce two data augmentation strategies and present evidence that the model learns generalizable and semantically meaningful information.

show abstract

Towards Linking the Lakh and IMSLP Datasets

Cited by 11 publications

References 8 publications

Exploiting Temporal Dependencies for Cross-Modal Music Piece Identification

Exploiting Temporal Dependencies for Cross-Modal Music Piece Identification

Automatic Generation of Piano Score Following Videos

A Deeper Look at Sheet Music Composer Classification Using Self-Supervised Pretraining

Contact Info

Product

Resources

About