MidiBERT-Piano: Large-scale Pre-training for Symbolic Music Understanding

Abstract

This paper presents an attempt to employ the mask language modeling approach of BERT to pre-train a 12-layer Transformer model over 4,166 pieces of polyphonic piano MIDI files for tackling a number of symbolic-domain discriminative music understanding tasks. These include two note-level classification tasks, i.e., melody extraction and velocity prediction, as well as two sequence-level classification tasks, i.e., composer classification and emotion classification. We find that, given a pretrained Transformer, our models outperform recurrent neural network based baselines with less than 10 epochs of fine-tuning. Ablation studies show that the pre-training remains effective even if none of the MIDI data of the downstream tasks are seen at the pre-training stage, and that freezing the self-attention layers of the Transformer at the fine-tuning stage slightly degrades performance. All the five datasets employed in this work are publicly available, as well as checkpoints of our pre-trained and fine-tuned models. As such, our research can be taken as a benchmark for symbolic-domain music understanding.

MusIAC: An extensible generative framework for Music Infilling Applications with multi-level Control

“…Firstly, music grammar is learned in the pretraining stage and then specific tasks are learned in the finetuning stage. This process is similar to the work of [8,33,6].…”

Conditional Drums Generation using Compound Word Representations

“…This strategy improves the efficiency of Transformer-based [42] architectures due to the decreased input sequence length which reduces the computational complexity. Recent studies show that CP achieves a better output quality compared to the aforementioned representations in certain tasks such as conditional/unconditional piano generation [18,8] and emotion recognition [21].…”

Beijing Qihoo Technology Co., Ltd. v. Tencent Technology (Shenzhen) Co., Ltd. and Shenzhen Tencent Computer System Co., Ltd. (Dispute over the Abuse of Market Dominant Position)—Analysis Methods and Ideas for the Definition of the Relevant Markets and the Abuse of Market Dominant Position in the Internet Environment

“…This approach offers simplicity and efficiency. Drawing inspiration from the remarkable achievements of BERT, Chou et al [5] introduced MidiBERTPiano, a large-scale pre-trained model utilizing CP representation. The proposed model showcases promising outcomes in various domains, including symbolic music emotion recognition.…”

“…Existing researches mainly apply deep-learning-based methods on the acoustic domain or uses sequencemodeling methods on the symbolic domain representations of the music. In their recent publication on emotion recognition in symbolic music, Qiu et al [31] introduced a pioneering approach utilizing the MIDIBERT model [4], a large-scale pre-trained music understanding model. At present, no existing research on Music Emotion Recognition (MER) for instrumental music integrates both acoustic and symbolic analyses.…”