I Am an Earphone and I Can Hear My User’s Face: Facial Landmark Tracking Using Smart Earphones

Abstract

This paper presents EARFace , a system that shows the feasibility of tracking facial landmarks for 3D facial reconstruction using in-ear acoustic sensors embedded within smart earphones. This enables a number of applications in the areas of facial expression tracking, user-interfaces, AR/VR applications, affective computing, accessibility, etc. While conventional vision-based solutions break down under poor lighting, occlusions, and also suffer from privacy concerns, earphone platforms are robust to ambient conditions, while being privacy-preserving. In contrast to prior work on earable platforms that perform outer-ear sensing for facial motion tracking, EARFace shows the feasibility of completely in-ear sensing with a natural earphone form-factor, thus enhancing the comfort levels of wearing. The core intuition exploited by EARFace is that the shape of the ear canal changes due to the movement of facial muscles during facial motion. EARFace tracks the changes in shape of the ear canal by measuring ultrasonic channel frequency response (CFR) of the inner ear, ultimately resulting in tracking of the facial motion. A transformer based machine learning (ML) model is designed to exploit spectral and temporal relationships in the ultrasonic CFR data to predict the facial landmarks of the user with an accuracy of 1.83 mm. Using these predicted landmarks, a 3D graphical model of the face that replicates the precise facial motion of the user is then reconstructed. Domain adaptation is further performed by adapting the weights of layers using a group-wise and differential learning rate. This decreases the training overhead in EARFace . The transformer based ML model runs on smartphone devices with a processing latency of 13 ms and an overall low power consumption profile. Finally, usability studies indicate higher levels of comforts of wearing EARFace ’s earphone platform in comparison with alternative form-factors.

Visualization Analysis of Smart Contract Technology Based on Citespace

“…Innovations in this space include EarFS [33], which uses electrodes in the ear canal to detect facial gestures, and TexonMask [19], which integrates textile electrodes in facemasks for expression recognition. Systems such as EARFace [64] and EarIO [26] demonstrate the use of acoustic sensors in earphones to track facial movements, offering privacy-preserving solutions. Additionally, newer technologies like millimeter wave radar have been explored for their potential to perform non-intrusive and privacy-preserving facial reconstructions, as seen in studies like mmFER [68] and mm3DFace [59].…”

“…The result is a high-frequency AU sequence output, operating at 30 frames per second, which ensures real-time responsiveness and utility for downstream applications. C-Face [12] Ear-mounted Camera ✗ ✗ ✗ ✗ NeckFace [11] Neck-mounted IR cameras ✗ ✓ ✗ ✓ FaceRecGlasses [1] Glasses Camera ✓ ✗ ✗ ✗ TexonMask [19] Mask Textile Electrodes ✗ ✓ ✓ ✗ EARFace [64] Ear-mounted Ultrasonic ✓ ✓ ✗ ✓ EarIO [26] Ear-mounted Microphone and speaker ✓ ✗ ✓ ✓ ExpressEar [54] Ear-mounted IMU-Only…”