RGB-D-T Based Face Recognition

Abstract

Facial images are of critical importance in many real-world applications from gaming to surveillance. The current literature on facial image analysis, from face detection to face and facial expression recognition, are mainly performed in either RGB, Depth (D), or both of these modalities. But, such analyzes have rarely included Thermal (T) modality. This paper paves the way for performing such facial analyzes using synchronized RGB-D-T facial images by introducing a database of 51 persons including facial images of different rotations, illuminations, and expressions. Furthermore, a face recognition algorithm has been developed to use these images. The experimental results show that face recognition using such three modalities provides better results compared to face recognition in any of such modalities in most of the cases.

Improved RGB‐D‐T based face recognition

“…This amounts to 900 images for each subject and a total of 45900 images for the whole dataset. During the experiments we have used the same sample splitting protocol between training and testing samples as provided in [18]. Training was performed using 2, 5, 10 or 25 evenly spaced samples.…”

“…Following the block diagram of the proposed system, shown in Figure 1, these hand-crafted features are extracted from each modality and independently normalized and, in the case of 6 HOGOM, reduced. These features are then concatenated into a single feature vector for training the Weighted Nearest Neighbour Classifier (WNNC) identical to the one in [18]. At the same time, each modality is also processed by modality-specific CNN, performing a stratified sampling and augmentation at each training epoch.…”

“…For extracting the HOGOM descriptor, faces were cropped using the ground truth provided in the dataset [18]. Ground truth data includes parameters of the face bounding box, which is based on the depth images.…”

“…Similar approaches for different, but related problems, like full body gender recognition have also been proposed [33]. Recently a database containing synchronised RGB, Depth, and Thermal (hereafter, RGB-D-T) images has been introduced in [18], including facial images of different rotations, illuminations, and expressions. The authors have highlighted the complementarity of different modalities motivating further research in this direction.…”

“…• The above mentioned fusions of facial features, have been applied to the multimodal RGB-D-T facial database of [18], highlighting the effect of the proposed feature fusion in advancing the recognition results.…”