Acoustic-based Upper Facial Action Recognition for Smart Eyewear

Abstract

Smart eyewear (e.g., AR glasses) is considered to be the next big breakthrough for wearable devices. The interaction of state-of-the-art smart eyewear mostly relies on the touchpad which is obtrusive and not user-friendly. In this work, we propose a novel acoustic-based upper facial action (UFA) recognition system that serves as a hands-free interaction mechanism for smart eyewear. The proposed system is a glass-mounted acoustic sensing system with several pairs of commercial speakers and microphones to sense UFAs. There are two main challenges in designing the system. The first challenge is that the system is in a severe multipath environment and the received signal could have large attenuation due to the frequency-selective fading which will degrade the system's performance. To overcome this challenge, we design an Orthogonal Frequency Division Multiplexing (OFDM)-based channel state information (CSI) estimation scheme that is able to measure the phase changes caused by a facial action while mitigating the frequency-selective fading. The second challenge is that because the skin deformation caused by a facial action is tiny, the received signal has very small variations. Thus, it is hard to derive useful information directly from the received signal. To resolve this challenge, we apply a time-frequency analysis to derive the time-frequency domain signal from the CSI. We show that the derived time-frequency domain signal contains distinct patterns for different UFAs. Furthermore, we design a Convolutional Neural Network (CNN) to extract high-level features from the time-frequency patterns and classify the features into six UFAs, namely, cheek-raiser, brow-raiser, brow-lower, wink, blink and neutral. We evaluate the performance of our system through experiments on data collected from 26 subjects. The experimental result shows that our system can recognize the six UFAs with an average F1-score of 0.92.

EyeEcho: Continuous and Low-power Facial Expression Tracking on Glasses

“…Prior work [6,34,72,82] have demonstrated that partial skin and muscle deformations behind and inside ears are highly informative to reconstruct full facial expressions when they are captured by different kinds of sensors. Xie et al [74] proved that the skin and muscle deformations around the eyes and the cheeks contain information that can be extracted to recognize upper facial gestures. The sensing hypothesis of EyeEcho is that these deformations around eyes and cheeks are highly informative about detailed facial movements on both lower and upper face including eyes, eyebrows, cheeks, and mouth.…”

“…To explore these research questions, we developed EyeEcho using active acoustic sensing to capture the skin deformations on glasses, which we will detail later. We chose acoustic sensing because its sensors are small, lightweight, low-power and have been successfully applied to various tasks on tracking human activities, including health-related activities detection [45,65], novel interaction methods [3,66,77,78], silent speech recognition [61,80,81,83], authentication [11,18,23,40,67], discrete facial expression recognition [74], gaze tracking [33], finger tracking [46,60], hand gesture recognition [31,79], body pose estimation [41], and motion tracking [35,84].…”

“…Existing glasses-based methods can only recognize discrete facial gestures. For instance, recent work on glasses using the acoustic-based method can only recognize 6 discrete upper facial expressions [74]. Second, smart glasses often come with limited battery capacity due to the weight restriction.…”

ExpresSense: Exploring a Standalone Smartphone to Sense Engagement of Users from Facial Expressions Using Acoustic Sensing

“…Smart glasses, coupled with microphones and speakers, have also been used for acoustic sensing. In [49], smart glasses are used to capture facial muscle movements and classify facial actions like cheek and brow raise, winks, etc.…”

InMyFace: Inertial and Mechanomyography-Based Sensor Fusion for Wearable Facial Activity Recognition

“…The wearable community has explored facial expression monitoring with different sensing modalities embedded in accessories. The various sensing methods include cameras [16,35], IMU [31,20], light [22], capacitive [39], piezoelectric [40], electromyography (EMG) [41], mechanomyography [28,2], audio based solutions [2,33,42], and many others. Table 1 shows a comparison of the state-of-the-art approaches.…”