Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio

Ferreira, João Pedro Moreira; Coutinho, Thiago M.; Gomes, Thiago L.; Neto, José Francisco dos Reis; Azevedo, Rafael Alves de; Martins, Renato; Nascimento, Erickson R.

doi:10.1016/j.cag.2020.09.009

Cited by 60 publications

(25 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Considering that the motion manifolds of classical convolutional and recursive neural models are non‐Euclidean geometry, Ferreira et al . [FCG*21] design a novel method based on graph convolutional networks to synthesize human motion from music. The goal of our research is different from the above work.…”

Section: Related Workmentioning

confidence: 99%

Diverse Dance Synthesis via Keyframes with Transformer Controllers

Pan

Wang

Bai

et al. 2021

Computer Graphics Forum

View full text Add to dashboard Cite

Existing keyframe-based motion synthesis mainly focuses on the generation of cyclic actions or short-term motion, such as walking, running, and transitions between close postures. However, these methods will significantly degrade the naturalness and diversity of the synthesized motion when dealing with complex and impromptu movements, e.g., dance performance and martial arts. In addition, current research lacks fine-grained control over the generated motion, which is essential for intelligent human-computer interaction and animation creation. In this paper, we propose a novel keyframe-based motion generation network based on multiple constraints, which can achieve diverse dance synthesis via learned knowledge. Specifically, the algorithm is mainly formulated based on the recurrent neural network (RNN) and the Transformer architecture. The backbone of our network is a hierarchical RNN module composed of two long short-term memory (LSTM) units, in which the first LSTM is utilized to embed the posture information of the historical frames into a latent space, and the second one is employed to predict the human posture for the next frame. Moreover, our framework contains two Transformer-based controllers, which are used to model the constraints of the root trajectory and the velocity factor respectively, so as to better utilize the temporal context of the frames and achieve fine-grained motion control. We verify the proposed approach on a dance dataset containing a wide range of contemporary dance. The results of three quantitative analyses validate the superiority of our algorithm. The video and qualitative experimental results demonstrate that the complex motion sequences generated by our algorithm can achieve diverse and smooth motion transitions between keyframes, even for long-term synthesis.

show abstract

Section: Related Workmentioning

confidence: 99%

Diverse Dance Synthesis via Keyframes with Transformer Controllers

Pan

Wang

Bai

et al. 2021

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…We also conducted a user study with 60 users. As discussed in [Ferreira et al 2020], we observed that most users could not distinguish between the real and generated motions in the user study. Moreover, we present the results for the most commonly used metric for the evaluation of generative models (Fréchet inception distance -FID) in Table 1.…”

Section: Experiments and Resultsmentioning

confidence: 79%

“…The results of this thesis were published into the international journal publication in the Computers & Graphics [Ferreira et al 2020]. This journal has been recently ranked 4 in the top Computer Graphics publications 2 .…”

Section: Publicationsmentioning

confidence: 99%

Synthesizing Realistic Human Dance Motions Conditioned by Musical Data using Graph Convolutional Networks

Ferreira¹,

Martins

Nascimento³

2021

Anais Do XXXIV Concurso De Teses E Dissertações Da SBC (CTD-SBC 2021)

Self Cite

View full text Add to dashboard Cite

Learning to move naturally from music, i.e., to dance, is one of the most complex motions humans often perform effortlessly. Existing techniques of automatic dance generation with classical CNN and RNN models undergo training and variability issues due to the non-Euclidean geometry of the motion manifold. We design a novel method based on GCNs to tackle the problem of automatic dance generation from audio. Our method uses an adversarial learning scheme conditioned on the input music audios to create natural motions. The results demonstrate that the proposed GCN model outperforms the state-of-the-art in different experiments. Visual results of the motion generation and explanation can be visualized through the link: http://youtu.be/fGDK6UkKzvA

show abstract

“…Similarly, the synthesis of human movement through learning technology is becoming an increasingly popular method in training instruction to alleviate the need for new data capture to produce animation (Ferreira, 2021). Dancers learn to move naturally from music.…”

Section: Audio Applied In Dance Teachingmentioning

confidence: 99%

Application and Development of Multimedia Technology in Dance in China

Sun¹

2022

Advances in Social Science, Education and Humanities Research

View full text Add to dashboard Cite

As the economy and technology continue to develop, technology, and multimedia are being used extensively in music, art, and dance. In particular, during the Covid-19 period, dance as an offline art was hit hard by isolation and social distance. Dancers and dance teachers have increased their internet, and digital media use to strengthen the links between dances. In addition, as digital media develops, the connection with dance will inevitably increase. Therefore, it is essential to analyse the application and development of multimedia technology in dance to take stock of and reflect on the current form of dance. Prospects and presuppositions are made in the form of dance. The result showed that by analyzing the application of multimedia technology in dance, the method of using multimedia technology was explored in dance performance and dance education, including the possible application of videos and projections, virtual reality and interactive installation in dance performances and choreography. In dance education, video, online classes, and audio provided the developmental scope of dance teaching. This research summarizes and prospects the intersection of dance and multimedia technology disciplines, lists the standard choreography methods of multimedia technology and dance, and teaches models that provide valuable thinking about the future development of dance.

show abstract

Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio

Cited by 60 publications

References 38 publications

Diverse Dance Synthesis via Keyframes with Transformer Controllers

Diverse Dance Synthesis via Keyframes with Transformer Controllers

Synthesizing Realistic Human Dance Motions Conditioned by Musical Data using Graph Convolutional Networks

Application and Development of Multimedia Technology in Dance in China

Contact Info

Product

Resources

About