Digital Facial Augmentation for Interactive Entertainment

Hieda, Naoto; Cooperstock, Jeremy R.

doi:10.4108/icst.intetain.2015.259444

Cited by 3 publications

(1 citation statement)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In these examples the facial motion is almost exclusively rigid, and retro‐reflective markers were required to be attached in advance. Most recently, Hieda et al [HC15] proposed a markerless, depth‐sensor based approach to interactively paint on a user's face using projection. This method, like all previous ones, does not aim at imperceptible latency for convincing augmentations.…”

Section: Related Workmentioning

confidence: 99%

Makeup Lamps: Live Augmentation of Human Faces via Projection

Bermano

Billeter

Iwai

et al. 2017

Computer Graphics Forum

View full text Add to dashboard Cite

Figure 1: Our system captures a live performance in the infrared (IR) spectrum (bottom left). A target appearance (top left) is rendered and saved as albedo and offsets based on expression and spatial position (top middle). These are blended and deformed to match the facial configuration and position (bottom middle), and projected back on the performer for appearance augmentation (right). The average system latency is 9.8ms, achieved through GPU optimizations, and is further compensated for through prediction. AbstractWe propose the first system for live dynamic augmentation of human faces. Using projector-based illumination, we alter the appearance of human performers during novel performances. The key challenge of live augmentation is latency -an image is generated according to a specific pose, but is displayed on a different facial configuration by the time it is projected. Therefore, our system aims at reducing latency during every step of the process, from capture, through processing, to projection. Using infrared illumination, an optically and computationally aligned high-speed camera detects facial orientation as well as expression. The estimated expression blendshapes are mapped onto a lower dimensional space, and the facial motion and non-rigid deformation are estimated, smoothed and predicted through adaptive Kalman filtering. Finally, the desired appearance is generated interpolating precomputed offset textures according to time, global position, and expression. We have evaluated our system through an optimized CPU and GPU prototype, and demonstrated successful low latency augmentation for different performers and performances with varying facial play and motion speed. In contrast to existing methods, the presented system is the first method which fully supports dynamic facial projection mapping without the requirement of any physical tracking markers and incorporates facial expressions. object or a person would look like when painted or altered in different ways, both statically and dynamically. For physical objects, this can be done with the aid of projection-based illumination, bringing the flexibly of digital objects to physical appearance [RWF99, BRF01, BBG * 13, AIS15, NWI15, ZXT * 16]. Since introduced, these methods have been used extensively in education, training, entertainment and sales.

show abstract

Section: Related Workmentioning

confidence: 99%