Recognition and mapping of facial expressions to avatar by embedded photo reflective sensors in head mounted display

Suzuki, Katsuhiro; Nakamura, Fumihiko; Otsuka, Jiu; Masai, Katsutoshi; Itoh, Yuta; Sugiura, Yuta; Sugimoto, Maki

doi:10.1109/vr.2017.7892245

Cited by 42 publications

(28 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The VR HMD occludes a significant portion of the user's face, preventing most existing facial performance capture approaches from being applied. Some recent works embedded small advanced optical sensors inside the headset, such as IR sensors [5], [8] to recognize facial expressions of basic emotions by detecting facial movements. Meanwhile, the contact-based sensing technology is drawing great attention in facial wearable device application because of its superiority in scenarios with highly constrained visibility.…”

Section: A Hmd-based Facial Sensing Systemsmentioning

confidence: 99%

“…Although existing techniques and approaches have pushed the boundary of HMD-based facial sensing systems forward, the following three problems remain unresolved: 1) Most previous systems [5], [8], [10], [11] concentrate on recognition of facial expression categories while ignoring the fact that facial expression has multiple appearance representations. For example, happiness can be expressed with either the AU12 (lip corner puller) or a combination of AU12 and AU6 (cheek raise), so a deeper insight into the composition (e.g.…”

Section: A Hmd-based Facial Sensing Systemsmentioning

confidence: 99%

“…facial performance sensing methods, such as ordinary camerabased technologies. A few recent works [5], [6], [7], [8], [9] have explored solutions to this problem. Li et al [6] and Olszewski et al [7] made preliminary trials of equipping HMDs with the facial performance capture ability.…”

mentioning

confidence: 99%

“…However, their solutions require a RGB-D or RGB camera mounted on the HMD, which are not ergonomically comfortable and cause an extra head burden. Some works resorted to other advanced facial sensing technologies, such as infrared (IR) sensors [5] and [8] electromyography (EMG) sensors [10], [11]. These lightweight optical or contact-based sensors can be easily embedded into the headset in an unobtrusive manner, thus open a new era of HMD-based wearable facial performance sensing systems.…”

mentioning

confidence: 99%

“…However, existing solutions [6], [7], [8] build the HMD user's face embodiment with non-realistic facial shape or texture. Moreover, a few systems [8] can only detect the facial expression category which is subsequently represented with pre-defined facial movements on a pre-made virtual avatar. This prohibits natural interactions between participants in the virtual world.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Realistic Facial Expression Reconstruction for VR HMD Users

Lou

Wang

Nduka³

et al. 2020

IEEE Trans. Multimedia

View full text Add to dashboard Cite

We present a system for sensing and reconstructing facial expressions of the virtual reality (VR) head-mounted display (HMD) user. The HMD occludes a large portion of the user's face, which makes most existing facial performance capturing techniques intractable. To tackle this problem, a novel hardware solution with electromyography (EMG) sensors being attached to the headset frame is applied to track facial muscle movements. For realistic facial expression recovery, we first reconstruct the user's 3D face from a single image and generate the personalized blendshapes associated with seven facial action units (AUs) on the most emotionally salient facial parts (ESFPs). We then utilize preprocessed EMG signals for measuring activations of AU-coded facial expressions to drive pre-built personalized blendshapes. Since facial expressions appear as important nonverbal cues of the subject's internal emotional states, we further investigate the relationship between six basic emotions -anger, disgust, fear, happiness, sadness and surprise, and detected AUs using a fern classifier. Experiments show the proposed system can accurately sense and reconstruct high-fidelity common facial expressions while providing useful information regarding the emotional state of the HMD user.

show abstract

Section: A Hmd-based Facial Sensing Systemsmentioning

confidence: 99%

Section: A Hmd-based Facial Sensing Systemsmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Realistic Facial Expression Reconstruction for VR HMD Users

Lou

Wang

Nduka³

et al. 2020

IEEE Trans. Multimedia

View full text Add to dashboard Cite

show abstract

Is the avatar scared? Pupil as a perceptual cue

Dong

Jörg

Jain

2022

Computer Animation & Virtual

View full text Add to dashboard Cite

The importance of eyes for virtual characters stems from the intrinsic social cues in a person's eyes. While previous work on computer generated eyes has considered realism and naturalness, there has been little investigation into how details in the eye animation impact the perception of an avatar's internal emotional state. We present three large scale experiments (N≈500) that investigate the extent to which viewers can identify if an avatar is scared. We find that participants can identify a scared avatar with 75% accuracy using cues in the eyes including pupil size variation, gaze, and blinks. Because eye trackers return pupil diameter in addition to gaze, our experiments inform practitioners that animating the pupil correctly will add expressiveness to a virtual avatar with negligible additional cost. These findings also have implications for creating expressive eyes in intelligent conversational agents and social robots.

show abstract

In‐panel ambient light eliminating differential circuit applied to active pixel fingerprint sensor

Tai,

Yuan,

Chen

et al. 2024

J Soc Info Display

View full text Add to dashboard Cite

In this paper, we present a novel approach to counter the influence of ambient light on photodetectors used in applications like biometric recognition and environmental sensing. The proposed solution introduces a circuit‐based technique that utilizes signal differencing to subtract ambient light signals before they reach the integrated circuit (IC). The process involves row and column differential signals, akin to analog circuit differential amplifiers. Simulations validate the circuit's functionality, showing its effectiveness in reducing ambient light impact. However, image reconstruction after differencing introduces blurriness due to the accumulation of noise. An alternative bidirectional fusion method is suggested, resulting in a clearer representation of features without noise accumulation. This innovative approach promises to enhance photodetector performance in challenging lighting conditions for various applications.

show abstract

Recognition and mapping of facial expressions to avatar by embedded photo reflective sensors in head mounted display

Cited by 42 publications

References 19 publications

Realistic Facial Expression Reconstruction for VR HMD Users

Realistic Facial Expression Reconstruction for VR HMD Users

Is the avatar scared? Pupil as a perceptual cue

In‐panel ambient light eliminating differential circuit applied to active pixel fingerprint sensor

Contact Info

Product

Resources

About