Music-driven motion editing

Cardle, Marc; Brooks, Stephen; Barthe, Loïc; Hassan, Mo; Robinson, Peter

doi:10.1145/1242073.1242235

Cited by 5 publications

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Let’s all dance: Enhancing amateur dance motions

Zhou

Qiong

et al. 2023

Comp. Visual Media

View full text Add to dashboard Cite

Professional dance is characterized by high impulsiveness, elegance, and aesthetic beauty. In order to reach the desired professionalism, it requires years of long and exhausting practice, good physical condition, musicality, but also, a good understanding of choreography. Capturing dance motions and transferring them to digital avatars is commonly used in the film and entertainment industries. However, so far, access to high-quality dance data is very limited, mainly due to the many practical difficulties in capturing the movements of dancers, making it prohibitive for large-scale data acquisition. In this paper, we present a model that enhances the professionalism of amateur dance movements, allowing movement quality to be improved in both spatial and temporal domains. Our model consists of a dance-to-music alignment stage responsible for learning the optimal temporal alignment path between dance and music, and a dance-enhancement stage that injects features of professionalism in both spatial and temporal domains. To learn a homogeneous distribution and credible mapping between the heterogeneous professional and amateur datasets, we generate amateur data from professional dances taken from the AIST++ dataset. We demonstrate the effectiveness of our method by comparing it with two baseline motion transfer methods via thorough qualitative visual controls, quantitative metrics, and a perceptual study. We also provide temporal and spatial module analysis to examine the mechanisms and necessity of key components of our framework.

show abstract

Let’s all dance: Enhancing amateur dance motions

Zhou

Qiong

et al. 2023

Comp. Visual Media

View full text Add to dashboard Cite

show abstract

MusicFace: Music-driven expressive singing face synthesis

Liu,

Deng,

et al. 2023

Comp. Visual Media

View full text Add to dashboard Cite

It remains an interesting and challenging problem to synthesize a vivid and realistic singing face driven by music. In this paper, we present a method for this task with natural motions for the lips, facial expression, head pose, and eyes. Due to the coupling of mixed information for the human voice and backing music in common music audio signals, we design a decouple-and-fuse strategy to tackle the challenge. We first decompose the input music audio into a human voice stream and a backing music stream. Due to the implicit and complicated correlation between the two-stream input signals and the dynamics of the facial expressions, head motions, and eye states, we model their relationship with an attention scheme, where the effects of the two streams are fused seamlessly. Furthermore, to improve the expressivenes of the generated results, we decompose head movement generation in terms of speed and direction, and decompose eye state generation into short-term blinking and long-term eye closing, modeling them separately. We have also built a novel dataset, SingingFace, to support training and evaluation of models for this task, including future work on this topic. Extensive experiments and a user study show that our proposed method is capable of synthesizing vivid singing faces, qualitatively and quantitatively better than the prior state-of-the-art.

show abstract

Emotional remapping of music to facial animation

DiPaola

Arya

2006

Proceedings of the 2006 ACM SIGGRAPH Symposium on Videogames

View full text Add to dashboard Cite

We propose a method to extract the emotional data from a piece of music and then use that data via a remapping algorithm to automatically animate an emotional 3D face sequence. The method is based on studies of the emotional aspect of music and our parametric-based behavioral head model for face animation. We address the issue of affective communication remapping in general, i.e. translation of affective content (eg. emotions, and mood) from one communication form to another. We report on the results of our MusicFace system, which use these techniques to automatically create emotional facial animations from multiinstrument polyphonic music scores in MIDI format and a remapping rule set.

show abstract

Music-driven motion editing

Cited by 5 publications

References 7 publications

Let’s all dance: Enhancing amateur dance motions

Let’s all dance: Enhancing amateur dance motions

MusicFace: Music-driven expressive singing face synthesis

Emotional remapping of music to facial animation

Contact Info

Product

Resources

About