Handwritten Music Recognition for Mensural notation with convolutional recurrent neural networks

Abstract

Optical Music Recognition is the technology that allows computers to read music notation, which is also referred to as Handwritten Music Recognition when it is applied over handwritten notation. This technology aims at efficiently transcribing written music into a representation that can be further processed by a computer. This is of special interest to transcribe the large amount of music written in early notations, such as the Mensural notation, since they represent largely unexplored heritage for the musicological community. Traditional approaches to this problem are based on complex strategies with many explicit rules that only work for one particular type of manuscript. Machine learning approaches offer the promise of generalizable solutions, based on learning from just labelled examples. However, previous research has not achieved sufficiently acceptable results for handwritten Mensural notation. In this work we propose the use of deep neural networks, namely convolutional recurrent neural networks, which have proved effective in other similar domains such as handwritten text recognition. Our experimental results achieve, for the first time, recognition results that can be considered effective for transcribing handwritten Mensural notation, decreasing the symbol-level error rate of previous approaches from 25.7 % to 7.0 %.

Evaluating Simultaneous Recognition and Encoding for Optical Music Recognition

“…In the previous work on which our approach is based [6,7], the network outputs these sequences with an agnostic (graphical) codification, which does not follow any standard digital music representation, as it only represents the music symbols by physical features such as the shape and its position in the staff. However, we will train the network to output a semantic encoding directly.…”

“…There are other approaches which do not have this limitation, such as the seq2seq approach. However, they have proven to be much less effective than the CRNN/CTC model for OMR [7]. What is important here is that the aforementioned operation of the CRNN/CTC approach forces us to directly discard highly verbose encodings, such as MEI [16] or MusicXML [13], as output formats to be considered.…”

“…The model consists of a CRNN with a CTC loss function. The configuration is based on the implementation proposed by Calvo-Zaragoza et al [7], which refined an original model developed by Shi et al [35] to perform music recognition. Precisely, the model consists of three convolutional layers with a Leaky ReLU activation function and a max-pooling down-sampling.…”

Using Optical Music Recognition to Encode 17th-Century Music Prints

“…Further testing on the identical data sets using additional metrics is necessary before we might reach firmer conclusions in this respect. Since other studies involve a set of additional criteria that we have not investigated to date, it is also difficult to compare our results with recent systems that focus on early handwritten music notations [16,38]; again this is a potential area for future investigation with IntelliOMR.…”

“…Arising out of the HISPAMUS project, Rizo et al [50] describe an effective online tool, MUsic Recognition, Encoding, and Transcription (MuRET ), for managing the OMR and encoding of monophonic sources, including techniques signalled in [38]. Lastly, Calvo-Zaragoza et al [16] utilize a hybrid approach, using traditional algorithms for deskewing the image and identifying staves and normalising the visual area to process, followed by a recurrent convolutional neural network, resulting in "effective transcription" of handwritten mensural notation of 93% accuracy. Also notable is recent work that uses full CNNs on early square music notation to achieve a 96% accuracy in symbol identification [56].…”