3D head-talk: speech synthesis 3D head movement face animation

Yang, Daowu; Li, Ruihui; Yang, Qi; Peng, Yuyi; Huang, Xibei; Zou, Jing

doi:10.1007/s00500-023-09292-5

Cited by 5 publications

(4 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This requires a set of comprehensive approaches that need to be used throughout the development process. These strategies are aimed at enhancing different aspects of realism and immersion, contributing to the creation of a convincing and engaging virtual experience using realistic audio-visual fusion [109], facial expressions and head pose consistency [116], 3D facial animation [103,106], synchronized audio and high-resolution 3D sequences [114], coherent poses and motion matching speech [107,111], lip synchronization and realistic expressions [58,61,62,113], audio quality and spatial realism [60], highly realistic motion synthesis [63], emotion transfer and user perception [108], improved visual appearance [57], realistic representation and spatial alignment [59].…”

Section: Realism Implementationmentioning

confidence: 99%

“…The drive for more lifelike animations is further supported by Liu et al [116], where a focus on realistic facial expressions and head movements adds an enhanced sense of reality. Finally, Yang et al [109] use a comprehensive approach, interweaving audio-visual content and facial animation to create a more authentic virtual environment, achieving the desired reality enhancement. By replicating natural facial expressions and movements, an environment is created where users feel fully engaged and connected.…”

Section: Covered Criteria For Talking Head Implementationmentioning

confidence: 99%

“…However, it covers only two general criteria, in contrast to [57], where all five are considered. Apart from telepresence criteria, which is fully covered by [57,58,60,62,63,[107][108][109]111,112,[114][115][116], there are no other 3D talking head algorithms that fully meet the all VE criteria. This gives reason to focus on the pervasiveness of the application of talking head algorithms and the trend toward VE telepresence.…”

Section: Seamless Integration Within Svementioning

confidence: 99%

“…For each of the algorithms presented in Table4, the degree of convergence of its application in shared virtual environments (AR, VR, MR) was calculated using the criteria presented in Table3. Those algorithms that cover 100% the criteria for telepresence are colored in blue, and the rest are colored black[57,58,60,63,[106][107][108][109]111,116].…”

mentioning

confidence: 99%

See 3 more Smart Citations

Application of a 3D Talking Head as Part of Telecommunication AR, VR, MR System: Systematic Review

Christoff,

Neshov,

Tonchev

et al. 2023

Electronics

View full text Add to dashboard Cite

In today’s digital era, the realms of virtual reality (VR), augmented reality (AR), and mixed reality (MR) collectively referred to as extended reality (XR) are reshaping human–computer interactions. XR technologies are poised to overcome geographical barriers, offering innovative solutions for enhancing emotional and social engagement in telecommunications and remote collaboration. This paper delves into the integration of (AI)-powered 3D talking heads within XR-based telecommunication systems. These avatars replicate human expressions, gestures, and speech, effectively minimizing physical constraints in remote communication. The contributions of this research encompass an extensive examination of audio-driven 3D head generation methods and the establishment of comprehensive evaluation criteria for 3D talking head algorithms within Shared Virtual Environments (SVEs). As XR technology evolves, AI-driven 3D talking heads promise to revolutionize remote collaboration and communication.

show abstract

Section: Realism Implementationmentioning

confidence: 99%

Section: Covered Criteria For Talking Head Implementationmentioning

confidence: 99%

Section: Seamless Integration Within Svementioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Application of a 3D Talking Head as Part of Telecommunication AR, VR, MR System: Systematic Review

Christoff,

Neshov,

Tonchev

et al. 2023

Electronics

View full text Add to dashboard Cite

show abstract

A dance movement quality evaluation model using transformer encoder and convolutional neural network

2024

Sci Rep

View full text Add to dashboard Cite

The purpose of this study is to put forward a new evaluation model of dance movement quality to deal with the subjectivity and inconsistency in traditional evaluation methods. In view of the complexity and diversity of dance art and the widespread popularity of dance videos on social media, it is particularly urgent to develop an automatic and efficient tool for evaluating the quality of dance movements. Therefore, this study puts forward the Transformer Convolutional Neural Network with Dynamic and Static Streams (TransCNN-DSSS) model, which combines the analysis of dynamic flow and static flow, and makes use of the advantages of Transformer and Convolutional Neural Network (CNN) to deeply analyze and evaluate the dance movements. The core of the model is Quality Score Decoupling (QSD), which decouples and weights different quality dimensions of dance movements through attention mechanism, such as accuracy, fluency and expressiveness. Score Prediction module (SPM) uses Transformer network to further process the fused features, and outputs the final evaluation score through the full connection layer. In the experimental part, the TransCNN-DSSS model is trained and tested on the marked dance movement dataset. The performance of the model is evaluated by accuracy, recall and F1 score. The results show that the model has achieved 90% accuracy, 89% recall and F1 score of 0.90 in the task of evaluating the quality of dance movements. These results prove the effectiveness and reliability of the model. In addition, the adaptability test of the model in different dance styles also shows good generalization ability. The research contribution of this study is to put forward a new evaluation model of dance movement quality, which provides an objective and automatic evaluation tool for dance teaching, competition scoring and fans.

show abstract