Sculptor

Abstract

Recent years have seen growing interest in 3D human face modeling due to its wide applications in digital human, character generation and animation. Existing approaches overwhelmingly emphasized on modeling the exterior shapes, textures and skin properties of faces, ignoring the inherent correlation between inner skeletal structures and appearance. In this paper, we present SCULPTOR, 3D face creations with Skeleton Consistency Using a Learned Parametric facial generaTOR , aiming to facilitate the easy creation of both anatomically correct and visually convincing face models via a hybrid parametric-physical representation. At the core of SCULPTOR is LUCY, the first large-scale shape-skeleton face dataset in collaboration with plastic surgeons. Named after the fossils of one of the oldest known human ancestors, our LUCY dataset contains high-quality Computed Tomography (CT) scans of the complete human head before and after orthognathic surgeries, which are critical for evaluating surgery results. LUCY consists of 144 scans of 72 subjects (31 male and 41 female), where each subject has two CT scans taken pre- and post-orthognathic operations. Based on our LUCY dataset, we learned a novel skeleton consistent parametric facial generator, SCULPTOR, which can create unique and nuanced facial features that help define a character and at the same time maintain physiological soundness. Our SCULPTOR jointly models the skull, face geometry and face appearance under a unified data-driven framework by separating the depiction of a 3D face into shape blend shape, pose blend shape and facial expression blend shape. SCULPTOR preserves both anatomic correctness and visual realism in facial generation tasks compared with existing methods. Finally, we showcase the robustness and effectiveness of SCULPTOR in various fancy applications unseen before, like archaeological skeletal facial completion, bone-aware character fusion, skull inference from images, face generation with lipo-Level change and facial animations, etc.

Digital Human: Three-dimentional face model reconstruction based on Hifi3dface model with clinical two-dimentional photos

“…Additionally, the landmark reduction ranged from 0.2 mm to 2.4 mm. Qiu et al [23] reported an improvement in the reconstruction result using CBCT data, with a facial MSE of approximately 1.5mm, but the result had poor extrapolation performance, with an error of 2.68mm.…”

Local anatomically-constrained facial performance retargeting

“…This formulation was later adapted by Chandran et al [10] for facial performance retargeting. Qiu et al proposed SCULPTOR [37], a multi-identity joint morphable model of facial anatomy and skin learned from a database of computed tomography (CT) scans. Recently Choi et al proposed Animatomy [15], a muscle fiber based anatomical basis for animator friendly face modeling applications.…”

“…Since this structure does not exist in reality and is, therefore, not available for supervised learning, we formulate a learning framework where such rigidly deforming surface can be learnt only from the sparse set of anatomic constraints that can be computed between the skin and the underlying bones. As we will see in Section 5, learning this anatomic surface from data leads to several interesting applications in shape manipulation and performance retargeting that were previously challenging to obtain without expensive physical simulation [48] or extensive volumetric data capture [37].…”