Self-Supervised Music Motion Synchronization Learning for Music-Driven Conducting Motion Generation

Liu, Fan; Chen, De-Long; Zhou, Rui-Zhi; Yang, Shenglai; Xu, Feng

doi:10.1007/s11390-022-2030-z

Cited by 5 publications

(2 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We use a wide array of metrics, including mean squared error (MSE), fractional shape distance (FGD), beat consistency score (BC) and diversity, to evaluate the motion produced by Diffusion-Conductor. Thorough comparisons demonstrated that our model outperforms the previous GAN-based method (Liu et al 2022).…”

Section: Introductionmentioning

confidence: 88%

“…Advancements in AIGC technologies for human motion (Mourot et al 2022) have addressed the generation of various human motions such as speech gestures, dance movements, and instrumental motions in recent years, and researchers are now pivoting towards building AI conductors. Pioneered works of Virtu-alConductor (Chen et al 2021) and M 2 S-GAN (Liu et al 2022) demonstrated the promising possibilities of building such systems. These works take advantage of the Generative Adversarial Network (GAN) (Goodfellow et al 2020) to learn the probabilistic distribution of real conducting motion from a large-scale paired music-motion dataset.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Taming Diffusion Models for Music-Driven Conducting Motion Generation

Zhao,

Bai,

Chen

et al. 2023

AAAI-SS

View full text Add to dashboard Cite

Generating the motion of orchestral conductors from a given piece of symphony music is a challenging task since it requires a model to learn semantic music features and capture the underlying distribution of real conducting motion. Prior works have applied Generative Adversarial Networks (GAN) to this task, but the promising diffusion model, which recently showed its advantages in terms of both training stability and output quality, has not been exploited in this context. This paper presents Diffusion-Conductor, a novel DDIM-based approach for music-driven conducting motion generation, which integrates the diffusion model to a two-stage learning framework. We further propose a random masking strategy to improve the feature robustness, and use a pair of geometric loss functions to impose additional regularizations and increase motion diversity. We also design several novel metrics, including Frechet Gesture Distance (FGD) and Beat Consistency Score (BC) for a more comprehensive evaluation of the generated motion. Experimental results demonstrate the advantages of our model. The code is released at https://github.com/viiika/Diffusion-Conductor.

show abstract

Section: Introductionmentioning

confidence: 88%

Section: Introductionmentioning

confidence: 99%