Temporally Guided Music-to-Body-Movement Generation

Kao, Hsuan-Kai; Su, Li

doi:10.1145/3394171.3413848

Cited by 30 publications

(7 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some researchers describe manufacturing capabilities as design and manufacturing capabilities, ascertained manufacturing capabilities, and actual manufacturing capabilities. According to manufacturing tasks, piano automation, equipment, and the relationship between roles, the model of the solution model of manufacturing capability from task expectation to demand deployment and the relationship model of piano automation composition from capability to role are established, and the piano automation composition hierarchical configuration model is proposed [22][23][24].…”

Section: Related Workmentioning

confidence: 99%

Piano Automatic Composition and Quantitative Perception under the Data-Driven Architecture

Ren

2022

Journal of Sensors

View full text Add to dashboard Cite

This paper combines automatic piano composition with quantitative perception, extracts note features from the demonstration audio, and builds a neural network model to complete automatic composition. First of all, in view of the diversity and complexity of the data collected in the quantitative perception of piano automatic composition, the energy efficiency-related state data of the piano automatic composition operation is collected, carried out, and dealt with. Secondly, a perceptual data-driven energy efficient evaluation and decision-making method is proposed. This method is based on time series index data. After determining the time subjective weight through time entropy, the time dimension factor is introduced, and then the subjective time weight is adjusted by the minimum variance method. Then, we consider the impact of the perception period on the perception efficiency and accuracy, calculate and dynamically adjust the perception period based on the running data, consider the needs of the perception object in different scenarios, and update the perception object in real time during the operation. Finally, combined with the level weights determined by the data-driven architecture, the dynamic manufacturing capability index and energy efficiency index of the equipment are finally obtained. The energy efficiency evaluation of the manufacturing system of the data-driven architecture proves the feasibility and scientificity of the evaluation method and achieves the goal of it. The simulation experiment results show that it can reduce the perception overhead while ensuring the perception efficiency and accuracy.

show abstract

Section: Related Workmentioning

confidence: 99%

Piano Automatic Composition and Quantitative Perception under the Data-Driven Architecture

Ren

2022

Journal of Sensors

View full text Add to dashboard Cite

show abstract

“…To overcome this shortcoming and improve the pose dynamics estimation, the attention mechanism has been widely used by the existing generative methods. For instance, Kao et al developed a generativeattention-based network for synthesizing the movement of the skeleton of a violinist playing a particular piece of music [88]. This method first extracts the Mel-Frequency Cepstral Coefficients (MFCC) features of the audio.…”

Section: B Dance Choreography Generationmentioning

confidence: 99%

A Survey of Cross-Modal Visual Content Generation

Nazarieh,

Feng,

Awais

et al. 2024

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

Cross-modal content generation has become very popular in recent years. To generate high-quality and realistic content, a variety of methods have been proposed. Among these approaches, visual content generation has attracted significant attention from academia and industry due to its vast potential in various applications. This survey provides an overview of recent advances in visual content generation conditioned on other modalities, such as text, audio, speech, and music, with a focus on their key contributions to the community. In addition, we summarize the existing publicly available datasets that can be used for training and benchmarking cross-modal visual content generation models. We provide an in-depth exploration of the datasets used for audio-to-visual content generation, filling a gap in the existing literature. Various evaluation metrics are also introduced along with the datasets. Furthermore, we discuss the challenges and limitations encountered in the area, such as modality alignment and semantic coherence. Last, we outline possible future directions for synthesizing visual content from other modalities including the exploration of new modalities, and the development of multi-task multi-modal networks. This survey serves as a resource for researchers interested in quickly gaining insights into this burgeoning field.

show abstract

“…With the great success of deep neural networks, many researchers have addressed music-to-dance as a generation problem using learning-based techniques. Recent methods have employed auto encoderdecoders [41], LSTM [42][43][44][45][46], GANs [47,48], and transformers [49][50][51]. Even though some work [47,52] applies action units to further explore the correlations between pose and music, they still find it challenging to generate diverse, rhythmic, expressive dance motions.…”

Section: Music-driven Animationmentioning

confidence: 99%

MusicFace: Music-driven expressive singing face synthesis

Liu,

Deng,

et al. 2023

Comp. Visual Media

View full text Add to dashboard Cite

It remains an interesting and challenging problem to synthesize a vivid and realistic singing face driven by music. In this paper, we present a method for this task with natural motions for the lips, facial expression, head pose, and eyes. Due to the coupling of mixed information for the human voice and backing music in common music audio signals, we design a decouple-and-fuse strategy to tackle the challenge. We first decompose the input music audio into a human voice stream and a backing music stream. Due to the implicit and complicated correlation between the two-stream input signals and the dynamics of the facial expressions, head motions, and eye states, we model their relationship with an attention scheme, where the effects of the two streams are fused seamlessly. Furthermore, to improve the expressivenes of the generated results, we decompose head movement generation in terms of speed and direction, and decompose eye state generation into short-term blinking and long-term eye closing, modeling them separately. We have also built a novel dataset, SingingFace, to support training and evaluation of models for this task, including future work on this topic. Extensive experiments and a user study show that our proposed method is capable of synthesizing vivid singing faces, qualitatively and quantitatively better than the prior state-of-the-art.

show abstract

Temporally Guided Music-to-Body-Movement Generation

Cited by 30 publications

References 22 publications

Piano Automatic Composition and Quantitative Perception under the Data-Driven Architecture

Piano Automatic Composition and Quantitative Perception under the Data-Driven Architecture

A Survey of Cross-Modal Visual Content Generation

MusicFace: Music-driven expressive singing face synthesis

Contact Info

Product

Resources

About