Robust Facial Expression Recognition Using Near Infrared Cameras

Abstract

In human-human communication we use verbal, vocal and non-verbal signals to communicate with others. Facial expressions are a form of non-verbal communication, recognizing them helps to improve the human-machine interaction. This paper proposes a system for pose- and illumination-invariant recognition of facial expressions using near-infrared camera images and precise 3D shape registration. Precise 3D shape information of the human face can be computed by means of Constrained Local Models (CLM), which fits a dense model to an unseen image in an iterative manner. We used a multi-class SVM to classify the acquired 3D shape into different emotion categories. Results surpassed human performance and show poseinvariant performance. Varying lighting conditions can influence the fitting process and reduce the recognition precision. We built a near-infrared and visible light camera array to test the method with different illuminations. Results shows that the near-infrared camera configuration is suitable for robust and reliable facial expression recognition with changing lighting conditions.

Facial Expression Recognition Using Visible, IR, and MSX Images by Early and Late Fusion of Deep Learning Models

“…A noncontact methodology to classify human emotions through thermal images of the face was presented using histogram features obtained from facial patches fed to a support vector machine (SVM) [26]. Using constrained local models (CLMs), which iteratively fit a dense model to an unseen image, precise 3D shape information of the human face can be calculated [27]. Most conventional methods rely heavily on feature extraction and classification techniques with significant preprocessing.…”

Three-Stream Convolutional Neural Network with Squeeze-and-Excitation Block for Near-Infrared Facial Expression Recognition

“…However, these methods must extract facial expression features manually. Jeni et al [36] proposed a 3D-shape-information-based recognition technique and further proved that an NIR camera configuration is suitable for facial expressions under light-changing conditions. Wu et al [37] proposed a three-stream 3D convolutional network for NIR facial expression recognition, using a combination of global and local features, but did not consider assigning different weights to local features.…”

Advanced local motion patterns for macro and micro facial expression recognition

“…Some approaches [51,42] report performances on near infrared (NI) sequences to test their method in different light settings. Under various light settings available in the dataset, our method achieves better results than handcrafted approaches [14,47,48] and is competitive with regard to recent deep learning approaches [20,18,11].…”