Recognizing multiple persons' facial expressions using HMM based on automatic extraction of significant frames from image sequences

Abstract

A method that can be used for for recognizing facial expressions of multiple persons is proposed. I n this method, the condition of facial muscles is assigned to a hidden state of a HMM for each expression. Then, the probability of the state is updated according to a feature vector obtained from image processing. Image processing is performed in two steps. First, a velocity vector is estimated from every two successive frames b y using an optical flow algorithm. Then, a two-dimensional Fourier transform is applied to a velocity vector field at the regions around an eye and the mouth. The coefficients f o r lower frequencies are selected to f o r m a feature vector. A mixture densit y is used f o r approximating the output probability of the HMM so as to represent a variation in facial expressions among persons. To cope with the case when two expressions are displayed contiguously, the HMM computation is modified such that when the peak of a facial motion is detected, a new sequence of facial expressions is assumed to start from the previous frame with minimal facial motion. Experiments show that a mixture density is effective because recognition accuracy improves as the number of mixtures increases. In addition, the method correctly recognizes a facial expression that contiguously follows another one.

Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences

“…In reality, such segmentation is not available and, hence, there is a need to find an automatic way of segmenting face image sequences into the different facial expressions pictured. A way to achieve this has been proposed by Otsuka and Ohya [50] and Cohen et al [51]. To cope with cases where two facial expressions of emotion are displayed contiguously, Otsuka and Ohya applied a heuristic approach and modified the employed Hidden Markov model (HMM) computation such that when the peak of a facial motion is detected, the current emotional expression is assumed to start from the previous frame with minimal facial motion.…”

Machine Learning Methods for Social Signal Processing

“…With a few exceptions, most of the dynamic approaches to classification of facial expressions are based on the variants of Dynamic Bayesian Networks (DBN) (e.g., Hidden Markov Models (HMM) and Conditional Random Fields (CRF)). For example, (Otsuka & Ohya 1997, Shang & Chan 2009) trained independent HMMs for each emotion category, and then performed emotion categorization by comparing the likelihoods of the HMMs. In (Otsuka & Ohya 1997), the input features are based on velocity vectors computed using the optical flow algorithm, while the observation probability, corresponding to the hidden states in the HMMs, is modeled using mixtures of Gaussians in order to account better for variation in facial expressions of different subjects.…”

Robust facial expression recognition in the presence of rotation and partial occlusion

“…The channels in the figure are colour-coded as red, yellow, blue and green, respectively. These are associated with the four facial zones: brows; eyes; cheeks; and mouth, in A Hidden Markov Model (HMM) [88] is used to recognize the six prototypic facial expres-…”