Improving Bag-of-Visual-Words Towards Effective Facial Expressive Image Classification

Abstract

Bag-of-Visual-Words (BoVW) approach has been widely used in the recent years for image classification purposes. However, the limitations regarding optimal feature selection, clustering technique, the lack of spatial organization of the data and the weighting of visual words are crucial. These factors affect the stability of the model and reduce performance. We propose to develop an algorithm based on BoVW for facial expression analysis which goes beyond those limitations. Thus the visual codebook is built by using k-Means++ method to avoid poor clustering. To exploit reliable low level features, we search for the best feature detector that avoids locating a large number of keypoints which do not contribute to the classification process. Then, we propose to compute the relative conjunction matrix in order to preserve the spatial order of the data by coding the relationships among visual words. In addition, a weighting scheme that reflects how important a visual word is with respect to a given image is introduced. We speed up the learning process by using histogram intersection kernel by Support Vector Machine to learn a discriminative classifier. The efficiency of the proposed algorithm is compared with standard bag of visual words method and with bag of visual words method with spatial pyramid. Extensive experiments on the CK+, the MMI and the JAFFE databases show good average recognition rates. Likewise, the ability to recognize spontaneous and non-basic expressive states is investigated using the DynEmo database.

Recognizing Facial Expressions of Occluded Faces Using Convolutional Neural Networks

“…In the past few years, most works [2,6,8,11,13,16,[18][19][20][21]23,25,31,[34][35][36] have focused on building and training deep neural networks in order to achieve stateof-the-art results. Engineered models based on handcrafted features [1,14,29,30] Fig. 1.…”

“…Engineered models based on handcrafted features [1,14,29,30] Fig. 1. Images (of people wearing VR headsets) with corresponding Grad-CAM [28] explanation masks and labels from a VGG-face model trained on lower-half images.…”

Local Learning With Deep and Handcrafted Features for Facial Expression Recognition

“…Interestingly, the top scoring system in the 2013 FER Challenge is a deep convolutional neural network [34], while the best handcrafted model ranked only on the fourth place [15]. With only a few exceptions [1,32,33], most of the recent works on facial expression recognition are based on deep learning [2,9,10,13,14,17,21,22,24,23,26,28,38,39,40]. Some of these recent works [14,17,21,38,39] proposed to train an ensemble of convolutional neural networks for improved performance, while others [6,16] combined deep features with handcrafted features such as SIFT [25] or Histograms of Oriented Gradients (HOG) [8].…”

“…In the past few years, most works [2,9,10,13,14,17,21,22,24,23,26,28,34,38,39,40] have focused on building and training deep neural networks in order to achieve state-of-the-art results. Engineered models based on handcrafted features [1,15,32,33] have drawn very little attention, since such models usually yield less accurate results compared to deep learning models. In this paper, we show that we can surpass the current state-of-the-art systems by combining automatic features learned by convolutional neural networks (CNN) and handcrafted features computed by the bag-of-visual-words (BOVW) model, especially when we employ local learning in the training phase.…”

information technology; image recognition; machine learning; plant identification from photo images; online-system; SURF; SIFT; FREAK; BOVW

“…The length of these vectors for each of the images is different depending on the number of key points found, which is a problem for further application of machine learning methods for image classification. This problem is solved by the presenting of the image in the form of Bag-Of-Visual-Words (BOVW) [7], this model was implemented in work. After that, the Support Vector Machine algorithm is applied, which allows the identification of plants.…”