Transformer VAE: A Hierarchical Model for Structure-Aware and Interpretable Music Representation Learning

Jiang, Junyan; Xia, Gus; Carlton, Dave B.; Anderson, Chris; Miyakawa, Ryan

doi:10.1109/icassp40776.2020.9054554

Cited by 69 publications

(29 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, Choi et al [26] proposed a Transformerbased autoencoder that achieved global representation for the musical contexts of polyphonic piano performance data. Jiang et al [27] introduced a hierarchical Transformer VAE to learn context-sensitive melody representation with selfattention blocks, enabling the model to control the melodic and rhythmic contexts.…”

Section: B Transformer-based Music Generationmentioning

confidence: 99%

See 1 more Smart Citation

Translating Melody to Chord: Structured and Flexible Harmonization of Melody With Transformer

et al. 2022

View full text Add to dashboard Cite

Recent deep learning approaches for melody harmonization have achieved remarkable performance by overcoming the uneven chord distributions of music data. However, most of these approaches have not attempted to capture an original melodic structure and generate structured chord sequences with appropriate rhythms. Hence, we use a Transformer-based architecture that directly maps lower-level melody notes into a semantic higher-level chord sequence. In particular, we encode the binary piano roll of a melody into a note-based representation. Furthermore, we address the flexible generation of various chords with Transformer expanded with a VAE framework. We propose three Transformer-based melody harmonization models: 1) the standard Transformer-based model for the neural translation of a melody to chords (STHarm), 2) the variational Transformer model for learning the global representation of complete music (VTHarm), and 3) the regularized variational Transformer model for the controllable generation of chords (rVTHarm). Experimental results demonstrate that the proposed models generate more structured, diverse chord sequences than LSTM-based models.

show abstract

Section: B Transformer-based Music Generationmentioning

confidence: 99%

“…To the best of our knowledge, we are the first to apply the VNMT approach to music generation. In particular, our approach is different from previous music generation studies using the variational Transformer, which mostly served as an autoencoder [26], [27].…”

Section: Introductionmentioning

confidence: 99%

Translating Melody to Chord: Structured and Flexible Harmonization of Melody With Transformer

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Pop music transformer [49] adopts transformer-XL to leverage longer-range information along with a new data representation that expresses the rhythmic and harmonic structure of music. Transformer variational autoencoder [50] enables joint learning of local representation and global structure based on the hierarchical modeling. Harmony transformer [51] improves chord recognition to integrate chord segmentation with a non-autoregressive decoding method in the framework of musical harmony analysis.…”

Section: Global Structure-aware Language Modelmentioning

confidence: 99%

Global Structure-Aware Drum Transcription Based on Self-Attention Mechanisms

Ishizuka

Nishikimi

Yoshii

2021

Signals

View full text Add to dashboard Cite

This paper describes an automatic drum transcription (ADT) method that directly estimates a tatum-level drum score from a music signal in contrast to most conventional ADT methods that estimate the frame-level onset probabilities of drums. To estimate a tatum-level score, we propose a deep transcription model that consists of a frame-level encoder for extracting the latent features from a music signal and a tatum-level decoder for estimating a drum score from the latent features pooled at the tatum level. To capture the global repetitive structure of drum scores, which is difficult to learn with a recurrent neural network (RNN), we introduce a self-attention mechanism with tatum-synchronous positional encoding into the decoder. To mitigate the difficulty of training the self-attention-based model from an insufficient amount of paired data and to improve the musical naturalness of the estimated scores, we propose a regularized training method that uses a global structure-aware masked language (score) model with a self-attention mechanism pretrained from an extensive collection of drum scores. The experimental results showed that the proposed regularized model outperformed the conventional RNN-based model in terms of the tatum-level error rate and the frame-level F-measure, even when only a limited amount of paired data was available so that the non-regularized model underperformed the RNN-based model.

show abstract

“…Pop music transformer [49] adopts transformer-XL to leverage longer-range information along with a new data representation that expresses the rhythmic and harmonic structure of music. Transformer variational autoencoder [50] enables the joint learning of local representation and global structure based on the hierarchical modeling. Harmony transformer [51] improves chord recognition to integrate chord segmentation with a non-autoregressive decoding method in the framework of musical harmony analysis.…”

Section: Global Structure-aware Language Modelmentioning

confidence: 99%

Global Structure-Aware Drum Transcription Based on Self-Attention Mechanisms

Ishizuka¹,

Nishikimi²,

Yoshii³

2021

Preprint

View full text Add to dashboard Cite

This paper describes an automatic drum transcription (ADT) method that directly estimates a tatum-level drum score from a music signal, in contrast to most conventional ADT methods that estimate the frame-level onset probabilities of drums. To estimate a tatum-level score, we propose a deep transcription model that consists of a frame-level encoder for extracting the latent features from a music signal and a tatum-level decoder for estimating a drum score from the latent features pooled at the tatum level. To capture the global repetitive structure of drum scores, which is difficult to learn with a recurrent neural network (RNN), we introduce a self-attention mechanism with tatum-synchronous positional encoding into the decoder. To mitigate the difficulty of training the self-attention-based model from an insufficient amount of paired data and improve the musical naturalness of the estimated scores, we propose a regularized training method that uses a global structure-aware masked language (score) model with a self-attention mechanism pretrained from an extensive collection of drum scores. Experimental results showed that the proposed regularized model outperformed the conventional RNN-based model in terms of the tatum-level error rate and the frame-level F-measure, even when only a limited amount of paired data was available so that the non-regularized model underperformed the RNN-based model.

show abstract

Transformer VAE: A Hierarchical Model for Structure-Aware and Interpretable Music Representation Learning

Cited by 69 publications

References 3 publications

Translating Melody to Chord: Structured and Flexible Harmonization of Melody With Transformer

Translating Melody to Chord: Structured and Flexible Harmonization of Melody With Transformer

Global Structure-Aware Drum Transcription Based on Self-Attention Mechanisms

Global Structure-Aware Drum Transcription Based on Self-Attention Mechanisms

Contact Info

Product

Resources

About