Animatomy: an Animator-centric, Anatomically Inspired System for 3D Facial Modeling, Animation and Transfer

Choi, Byungkuk; Eom, Haekwang; Mouscadet, Benjamin; Cullingford, Stephen; Ma, Kurt; Gassel, Stefanie; Kim, Suzi; Moffat, Andrew; Maier, Millicent; Revelant, Marco; Letteri, Joe; Singh, Karan

doi:10.1145/3550469.3555398

Cited by 16 publications

(4 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By combining these shapes, diverse facial expressions are achieved, making blendshapes a prevalent technique in film and gaming [LAR*14]. In a recent work by Choi et al [CEM*22] is introduced the advanced facial animation system “Animatomy” which deviates from FACS‐blendshape systems, offering detailed, anatomically accurate control and easy animation transfer from actor to virtual character. In this survey, we will focus on contemporary systems widely utilized today, rather than an extensive discussion of facial expression capture research.…”

Section: Musical Performance Capturementioning

confidence: 99%

Virtual Instrument Performances (VIP): A Comprehensive Review

Kyriakou,

de la Campa Crespo,

Panayiotou

et al. 2024

Computer Graphics Forum

View full text Add to dashboard Cite

Driven by recent advancements in Extended Reality (XR), the hype around the Metaverse, and real‐time computer graphics, the transformation of the performing arts, particularly in digitizing and visualizing musical experiences, is an ever‐evolving landscape. This transformation offers significant potential in promoting inclusivity, fostering creativity, and enabling live performances in diverse settings. However, despite its immense potential, the field of Virtual Instrument Performances (VIP) has remained relatively unexplored due to numerous challenges. These challenges arise from the complex and multi‐modal nature of musical instrument performances, the need for high precision motion capture under occlusions including the intricate interactions between a musician's body and fingers with instruments, the precise synchronization and seamless integration of various sensory modalities, accommodating variations in musicians' playing styles, facial expressions, and addressing instrument‐specific nuances. This comprehensive survey delves into the intersection of technology, innovation, and artistic expression in the domain of virtual instrument performances. It explores musical performance multi‐modal databases and investigates a wide range of data acquisition methods, encompassing diverse motion capture techniques, facial expression recording, and various approaches for capturing audio and MIDI data (Musical Instrument Digital Interface). The survey also explores Music Information Retrieval (MIR) tasks, with a particular emphasis on the Musical Performance Analysis (MPA) field, and offers an overview of various works in the realm of Musical Instrument Performance Synthesis (MIPS), encompassing recent advancements in generative models. The ultimate aim of this survey is to unveil the technological limitations, initiate a dialogue about the current challenges, and propose promising avenues for future research at the intersection of technology and the arts.

show abstract

Section: Musical Performance Capturementioning

confidence: 99%

Virtual Instrument Performances (VIP): A Comprehensive Review

Kyriakou,

de la Campa Crespo,

Panayiotou

et al. 2024

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…Wētā FX proposed Animatomy [24], an end-to-end modular deformation architecture to approximately represent the deformation of muscle fibers and obtain anatomically plausible animation controls. The Animatomy system enables the automatic optimization of specific face parameters based on dynamic facial scans, animation driven by performance capture, dynamic simulation, and animation transfer.…”

Section: Animatomy By Wētā Fxmentioning

confidence: 99%

A survey on the pipeline evolution of facial capture and tracking for digital humans

Vilchis

Perez-Guerrero²,

Mendez-Ruiz³

et al. 2023

Multimedia Systems

View full text Add to dashboard Cite

With the introduction of concepts for virtual interaction and digital doubles, a rich scenario has been created for embodied avatars to strive. These avatars, more recently referred to as digital humans, have become a popular area of research, resulting in various techniques and methods that focus on improving the perception of their realism, fidelity, emphatic response, and interactivity. This survey aims to explore the literature and recent advancements on the key processes behind the creation and animation of digital human faces through the view of a general pipeline. The extensive review carried out in this study explores the usual data collection protocols, the main facial codification paradigms and databases, the approaches for digital human asset creation, facial tracking solutions for performance-driven animation, the solving process, and the final rendering delivery. Different quantitative evaluation methods, visual perception tests, and empathetic response evaluations for digital humans are also included in the survey. Additionally, the paper presents an updated summary of public and private frameworks for digital humans that go through the complete general pipeline presented. Finally, the condensed knowledge is discussed, inquiring into the possible direction of future developments in the field.

show abstract

“…Qiu et al proposed SCULPTOR [37], a multi-identity joint morphable model of facial anatomy and skin learned from a database of computed tomography (CT) scans. Recently Choi et al proposed Animatomy [15], a muscle fiber based anatomical basis for animator friendly face modeling applications. Lastly we recognize several physically based face models [39,41,44,48] which inherently have the ability to model anatomy constraints through simulation.…”

Section: Related Workmentioning

confidence: 99%

“…As they are ubiquitously used both in high-end production workflows and lightweight consumer applications, building expressive face models for various applications continues to remain an active area of research [17]. Face models today can range from simple linear global shape models [4,27,29] to highly complex local models that incorporate the underlying facial anatomy through physical simulation [15,44,48] or through anatomical constraints [47].…”

Section: Introductionmentioning

confidence: 99%

Local anatomically-constrained facial performance retargeting

et al. 2022

View full text Add to dashboard Cite

Generating realistic facial animation for CG characters and digital doubles is one of the hardest tasks in animation. A typical production workflow involves capturing the performance of a real actor using mo-cap technology, and transferring the captured motion to the target digital character. This process, known as retargeting , has been used for over a decade, and typically relies on either large blendshape rigs that are expensive to create, or direct deformation transfer algorithms that operate on individual geometric elements and are prone to artifacts. We present a new method for high-fidelity offline facial performance retargeting that is neither expensive nor artifact-prone. Our two step method first transfers local expression details to the target, and is followed by a global face surface prediction that uses anatomical constraints in order to stay in the feasible shape space of the target character. Our method also offers artists with familiar blendshape controls to perform fine adjustments to the retargeted animation. As such, our method is ideally suited for the complex task of human-to-human 3D facial performance retargeting, where the quality bar is extremely high in order to avoid the uncanny valley, while also being applicable for more common human-to-creature settings. We demonstrate the superior performance of our method over traditional deformation transfer algorithms, while achieving a quality comparable to current blendshape-based techniques used in production while requiring significantly fewer input shapes at setup time. A detailed user study corroborates the realistic and artifact free animations generated by our method in comparison to existing techniques.

show abstract

Animatomy: an Animator-centric, Anatomically Inspired System for 3D Facial Modeling, Animation and Transfer

Cited by 16 publications

References 39 publications

Virtual Instrument Performances (VIP): A Comprehensive Review

Virtual Instrument Performances (VIP): A Comprehensive Review

A survey on the pipeline evolution of facial capture and tracking for digital humans

Local anatomically-constrained facial performance retargeting

Contact Info

Product

Resources

About