Music-Driven Dance Generation

Qi, Yu; Liu, Yazhou; Sun, Quansen

doi:10.1109/access.2019.2953698

Cited by 16 publications

(8 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Due to the great amount of freely accessible dance videos, mostly hosted on platforms such as YouTube, multiple studies utilized automatic pose estimation methods to extract 2D skeletal poses from such videos and construct training data. The system presented in [19] proposed an autoregressive multimodal autoencoder based on two LSTM encoders for the two unimodal inputs (skeletal features and music features), which are fused in the decoder with a self-attention mechanism. Similarly, the authors in [18] proposed a multimodal convolutional autoencoder for synthesizing original dance sequences, conditioned on the mel-spectrogram of an input song.…”

Section: Related Workmentioning

confidence: 99%

“…Most of the proposed methods in the literature tackle the challenges of automatic dance motion generation by implementing Recurrent Neural Networks (RNNs) for modelling the temporal correlation of skeletal poses with musical information and generating novel motion sequences [13], [15], [19]- [26]. Nevertheless, RNNs are computationally cumbersome and inefficient for modelling very long sequences, since the error accumulation in the predicted pose sequences allows synthesis over a limited range of future time-steps [27].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

DanceConv: Dance Motion Generation With Convolutional Networks

et al. 2022

View full text Add to dashboard Cite

Automatically synthesizing dance motion sequences is an increasingly popular research task in the broader field of human motion analysis. Recent approaches have mostly used recurrent neural networks (RNNs), which are known to suffer from prediction error accumulation, usually limiting models to synthesize short choreographies of less than 100 poses. In this paper we present a multimodal convolutional autoencoder that combines 2D skeletal and audio information by employing an attention-based feature fusion mechanism, capable of generating novel dance motion sequences of arbitrary length. We first validate the ability of our system to capture the temporal context of dancing in a unimodal setting, by considering only skeletal features as input. According to 1440 rating answers provided by 24 participants in our initial user-study, we show that the optimal performance was presented by the model that was trained with input sequences of 500 poses. Based on this outcome, we train the proposed multimodal architecture with two different approaches, namely teacher-forcing and self-supervised curriculum learning, to deal with the autoregressive error accumulation phenomenon. In our evaluation campaign, we generate 1800 sequences and compare our method against two state-of-the-art approaches. Through qualitative and quantitative experiments we demonstrate the improvements introduced by the proposed multimodal architecture in terms of realism, motion diversity and multimodality, reducing the Fréchet Inception Distance (FID) metric value by 0.39. Subjective results confirm the effectiveness of our approach to synthesize diverse dance motion sequences, reporting a 6% increase in style consistency preference according to 1800 answers provided by 45 evaluators.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

DanceConv: Dance Motion Generation With Convolutional Networks

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Much work has been done in the area of automatic choreography generation from music. The work can be categorized into searching based methods [5], [13], [30]- [32], shallow neural networks like Factored Conditional Restricted Boltzmann Machine (FCRBM) [10], deep neural networks [7], [9], [11], [33], [34], and other methods including dynamic time warping [35] and hidden Markov models [6], [14].…”

Section: Music To Motion Generationmentioning

confidence: 99%

A Deep Music Recommendation Method Based on Human Motion Analysis

Gong

2021

IEEE Access

View full text Add to dashboard Cite

Despite its applications in automatic video editing and automatic music composition, the problem of music recommendation from dance motions has seldom been explored. In order to solve this problem, this work proposes a deep music recommendation algorithm based on dance motion analysis and evaluate it through quantitative measures. For quantitative evaluation, this work implements a LSTM-AE based music recommendation method which learns the correspondences between motion and music. In experiments, the two methodologies are compared and the motion analysis based methods outperform their rival by large margins. This work also proposes a quantitative measure of accurately recommended music genre. The proposed motion analysis based method achieves a recommendation accuracy of 91.3% using late fusion of joint and limb features.

show abstract

“…Mel frequency cepstral coefficients (MFCC) features and their derivatives are commonly used to describe the pitch of a song in music-driven animation generation, and have shown to produce good results (Fukayama and Goto, 2015;Alemi et al, 2017;Lee et al, 2018;Lee et al, 2018;Tang et al, 2018;Shlizerman et al, 2018;Qi et al, 2019). A Mel is a unit of measure based on how human ears perceive frequency.…”

Section: Pitchmentioning

confidence: 99%

“…Because the audio is recorded in a different frequency (sampling rate of 44100 kHz or 44100 audio samples per second) than the motion capture data (100 video frames per second), the MFCC features must be aligned to the motion capture by dividing the audio into windows of size h, which is computed by (Qi et al, 2019):…”

Section: Pitchmentioning

confidence: 99%