Using Cell Phone Pictures of Sheet Music To Retrieve MIDI Passages

Tsai, TJ; Yang, Daniel; Shan, Mengyi; Tanprasert, Thitaree; Jenrungrot, Teerapat

doi:10.1109/tmm.2020.2973831

Cited by 12 publications

(6 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For subsequent stave position detection, it is necessary to preprocess original sheet music photos. We use the same preprocessing method as [39]. Specifically, the steps for preprocessing are as follows:…”

Section: Preprocessingmentioning

confidence: 99%

A Stave-Aware Optical Music Recognition on Monophonic Scores for Camera-Based Scenarios

Liu,

Wu,

et al. 2023

Applied Sciences

View full text Add to dashboard Cite

The recognition of printed music sheets in camera-based realistic scenarios is a novel research branch of optical music recognition (OMR). However, special factors in realistic scenarios, such as uneven lighting distribution and curvature of staff lines, can have adverse effects on OMR models designed for digital music scores. This paper proposes a stave-aware method based on object detection to recognize monophonic printed sheet music in camera-based scenarios. By detecting the positions of staff lines, we improve the accuracy of note pitch effectively. In addition, we present the Camera Printed Music Staves (CPMS) dataset, which consists of labels and images captured by mobile phones under different angles and lighting conditions in realistic scenarios. We compare our method after training on different datasets with a sequence recognition method called CRNN-CTC on the test set of the CPMS dataset. The results show that the accuracy, robustness, and data dependency of our method perform better.

show abstract

Section: Preprocessingmentioning

confidence: 99%

A Stave-Aware Optical Music Recognition on Monophonic Scores for Camera-Based Scenarios

Liu,

Wu,

et al. 2023

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…This feature extraction includes the two upper left blocks in Figure 1, as well as the offline construction of the sheet music database. The bootleg score is a recently proposed feature representation for aligning piano sheet music images and MIDI (Yang et al, 2019;Tsai et al, 2020). It is a manually designed, low-dimensional feature representation that encodes the location of noteheads in sheet music relative to the staff lines.…”

Section: Feature Extractionmentioning

confidence: 99%

“…The data we used is derived from the Sheet Midi Retrieval (SMR) dataset (Tsai et al, 2020). The SMR dataset contains scanned sheet music from IMSLP and corresponding MIDI files for 200 solo piano pieces across 25 composers.…”

Section: Datamentioning

confidence: 99%

Automatic Generation of Piano Score Following Videos

Shan

Tsai

2021

Transactions of the International Society for Music Information Retrieval

Self Cite

View full text Add to dashboard Cite

This article studies the problem of generating a piano score following video from an audio recording in a fully automated manner. This problem contains two components: identifying the piece and aligning the audio with raw sheet music images. Unlike previous work, we focus primarily on working with raw, unprocessed sheet music from IMSLP, which may contain filler pages, other unrelated pieces or movements, or repeats and jumps whose locations are unknown a priori. To solve this problem, we combine state-ofthe-art methods with a novel alignment algorithm called Hierarchical DTW to handle discontinuities, which are the bottleneck on system performance. Hierarchical DTW handles repeats and jumps by considering all line breaks as possible jump locations, and it applies DTW at both a feature level and a segment level. We evaluate our algorithm with 200 PDFs from IMSLP and real audio recordings from Youtube. Our experiments show that Hierarchical DTW consistently outperforms a previously proposed Jump DTW algorithm in handling various types of discontinuities. We present extensive experimental results and analysis of the proposed algorithm.

show abstract

“…Composers of various periods have continuously enriched the diversification of creative techniques and creative methods through continuous exploration and induction of composition techniques. Today, the theory of composition technology has developed to a very complete level, and at the same time, this development process has also left its mark through a large number of excellent musical works accumulated in various periods [ 5 ]. Throughout the history of modern art development, it is actually a history of the evolution of art and technology, and the development of digital media art has also changed with the changes and development of the two.…”

Section: Introductionmentioning

confidence: 99%

Design of Neural Network Model for Cross-Media Audio and Video Score Recognition Based on Convolutional Neural Network Model

Liu

2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

In this paper, the residual convolutional neural network is used to extract the note features in the music score image to solve the problem of model degradation; then, multiscale feature fusion is used to fuse the feature information of different levels in the same feature map to enhance the feature representation ability of the model. A network composed of a bidirectional simple loop unit and a chained time series classification function is used to identify notes, parallelizing a large number of calculations, thereby speeding up the convergence speed of training, which also makes the data in the dataset no longer need to be strict with labels. Alignment also reduces the requirements on the dataset. Aiming at the problem that the existing cross-modal retrieval methods based on common subspace are insufficient for mining local consistency within modalities, a cross-modal retrieval method fused with graph convolution is proposed. The K-nearest neighbor algorithm is used to construct modal graphs for samples of different modalities, and the original features of samples from different modalities are encoded through a symmetric graph convolutional coding network and a symmetric multilayer fully connected coding network, and the encoded features are fused and input. We jointly optimize the intramodal semantic constraints and intermodal modality-invariant constraints in the common subspace to learn highly locally consistent and semantically consistent common representations for samples from different modalities. The error value of the experimental results is used to illustrate the effect of parameters such as the number of iterations and the number of neurons on the network. In order to more accurately illustrate that the generated music sequence is very similar to the original music sequence, the generated music sequence is also framed, and finally the music sequence spectrogram and spectrogram are generated. The accuracy of the experiment is illustrated by comparing the spectrogram and the spectrogram, and genre classification predictions are also performed on the generated music to show that the network can generate music of different genres.

show abstract

Using Cell Phone Pictures of Sheet Music To Retrieve MIDI Passages

Cited by 12 publications

References 36 publications

A Stave-Aware Optical Music Recognition on Monophonic Scores for Camera-Based Scenarios

A Stave-Aware Optical Music Recognition on Monophonic Scores for Camera-Based Scenarios

Automatic Generation of Piano Score Following Videos

Design of Neural Network Model for Cross-Media Audio and Video Score Recognition Based on Convolutional Neural Network Model

Contact Info

Product

Resources

About