Facial Muscle Activity Recognition with Reconfigurable Differential Stethoscope-Microphones

Abstract

Many human activities and states are related to the facial muscles’ actions: from the expression of emotions, stress, and non-verbal communication through health-related actions, such as coughing and sneezing to nutrition and drinking. In this work, we describe, in detail, the design and evaluation of a wearable system for facial muscle activity monitoring based on a re-configurable differential array of stethoscope-microphones. In our system, six stethoscopes are placed at locations that could easily be integrated into the frame of smart glasses. The paper describes the detailed hardware design and selection and adaptation of appropriate signal processing and machine learning methods. For the evaluation, we asked eight participants to imitate a set of facial actions, such as expressions of happiness, anger, surprise, sadness, upset, and disgust, and gestures, like kissing, winkling, sticking the tongue out, and taking a pill. An evaluation of a complete data set of 2640 events with 66% training and a 33% testing rate has been performed. Although we encountered high variability of the volunteers’ expressions, our approach shows a recall = 55%, precision = 56%, and f1-score of 54% for the user-independent scenario(9% chance-level). On a user-dependent basis, our worst result has an f1-score = 60% and best result with f1-score = 89%. Having a recall ≥60% for expressions like happiness, anger, kissing, sticking the tongue out, and neutral(Null-class).

InMyFace: Inertial and Mechanomyography-Based Sensor Fusion for Wearable Facial Activity Recognition

“…The wearable community has explored facial expression monitoring with different sensing modalities embedded in accessories. The various sensing methods include cameras [16,35], IMU [31,20], light [22], capacitive [39], piezoelectric [40], electromyography (EMG) [41], mechanomyography [28,2], audio based solutions [2,33,42], and many others. Table 1 shows a comparison of the state-of-the-art approaches.…”

“…MMG does not need an electrical connection with the skin and can be detected by various types of sensors, such as IMU, piezoelectric, microphones, pressure sensors, laser sensors, and many others [27]. In [2], facial muscle acoustic mechanomyography (AMMG) has been proposed. The authors used stethoscope microphones distributed around the temporalis, frontalis, and masseter muscles to detect ten facial expressions.…”

“…This work proposes a multimodal system that fuses complementary sensing modalities, such as inertial sensing IMU (orientation and acceleration), AMMG, and PMMG (force-sensitive resistor and piezoelectric film), integrated into a sports cap accessory. The facial expressions to be evaluated come from the Warsaw Photoset (seven expressions) and [2] (two facial movements); thus, it will be simpler to compare with future solutions, see Figure 1.…”

“…Our unique dataset demonstrates the inclusiveness and generalizability of the results. For the evaluation, we use the Warsaw Photoset [1] plus two facial gestures from [2].…”

Artificial Intelligence in Medicine: Real Time Electronic Stethoscope for Heart Diseases Detection

“…Recently, a large number of studies using data analysis have been conducted in cardiology. The accumulated data on electrocardiography and phonocardiography are analyzed for the correct diagnosis [8][9][10][11]. Auscultation of the heart continues to play an important diagnostic role in the assessment of heart health [12].…”