Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs

Abstract

To apply neural sequence models such as the Transformers to music generation tasks, one has to represent a piece of music by a sequence of tokens drawn from a finite set of pre-defined vocabulary. Such a vocabulary usually involves tokens of various types. For example, to describe a musical note, one needs separate tokens to indicate the note’s pitch, duration, velocity (dynamics), and placement (onset time) along the time grid. While different types of tokens may possess different properties, existing models usually treat them equally, in the same way as modeling words in natural languages. In this paper, we present a conceptually different approach that explicitly takes into account the type of the tokens, such as note types and metric types. And, we propose a new Transformer decoder architecture that uses different feed-forward heads to model tokens of different types. With an expansion-compression trick, we convert a piece of music to a sequence of compound words by grouping neighboring tokens, greatly reducing the length of the token sequences. We show that the resulting model can be viewed as a learner over dynamic directed hypergraphs. And, we employ it to learn to compose expressive Pop piano music of full-song length (involving up to 10K individual tokens per song), both conditionally and unconditionally. Our experiment shows that, compared to state-of-the-art models, the proposed model converges 5 to 10 times faster at training (i.e., within a day on a single GPU with 11 GB memory), and with comparable quality in the generated music

The Beauty of Repetition in Machine Composition Scenarios

“…Music Transformer [14] was the first Transformer-based model in symbolic music generation, showing that a Transformer model can generate coherent minute-long polyphonic piano music with reasonable repetitions and variance. Other Transformer-based models have since been proposed to generate music [9,13,15,17,24,28,36]. These models can learn musical features without hand-crafted rules and produce diverse music pieces.…”

“…We adopt the most established Transformer encoder architecture [31], which has been frequently applied in music generation as of late [13,37]. We follow the standard Transformer encoder architecture [31] and illustrate it in Figure 2 (b).…”

A Fuzzy Training Framework for Controllable Sequence-to-Sequence Generation

“…REMI [36] is a common representation format that uses [bar] and [position] tokens to place tokens on a metrical grid that uniformly divides a bar into a certain number of positions and assumes symbolic timing. Based on this, Hsiao et al [58] further employed an expansion-compression trick to convert a piece of music to a sequence of compound words by grouping neighboring tokens, significantly shortening the token sequences. As a result, we convert MIDI to the compound word format as our input.…”

“…As a result, we convert MIDI to the compound word format as our input. According to the practice of Hsiao et al [58], we convert MIDI into 7 types of symbols: Tempo, Chord, Bar-beat, Type, Pitch, Duration, and Velocity. For better illustration, we show part of melodies of Joe Hisaishi's "The Sun Also Rises" and transformed MIDI into compound words, as shown in Figure 2.…”

“…Based on the pretrained audio to midi transcription model provided by Kong et al [61], mp3 is transcribed into MIDI format. Then, we performed MP3-MIDI synchronization, melody extraction, chord recognition, and quantization steps following [58] to obtain normalized compound word format. We perform gap identification for music sequence segmentation according to the time information given in the lyrics.…”