Effective training of convolutional neural networks for age estimation based on knowledge distillation

Abstract

Age estimation from face images can be profitably employed in several applications, ranging from digital signage to social robotics, from business intelligence to access control. Only in recent years, the advent of deep learning allowed for the design of extremely accurate methods based on convolutional neural networks (CNNs) that achieve a remarkable performance in various face analysis tasks. However, these networks are not always applicable in real scenarios, due to both time and resource constraints that the most accurate approaches often do not meet. Moreover, in case of age estimation, there is the lack of a large and reliably annotated dataset for training deep neural networks. Within this context, we propose in this paper an effective training procedure of CNNs for age estimation based on knowledge distillation, able to allow smaller and simpler “student” models to be trained to match the predictions of a larger “teacher” model. We experimentally show that such student models are able to almost reach the performance of the teacher, obtaining high accuracy over the LFW+, LAP 2016 and Adience datasets, but being up to 15 times faster. Furthermore, we evaluate the performance of the student models in the presence of image corruptions, and we demonstrate that some of them are even more resilient to these corruptions than the teacher model.

Combining Different V1 Brain Model Variants to Improve Robustness to Image Corruptions in CNNs

“…Knowledge distillation has been shown to improve robustness to image corruptions [38,40,42,18]. Here, we demonstrate, for the first time, that a V1-inspired CNN can lead to higher robustness gains through distillation.…”

“…Ensembling is a well-known machine learning technique to combine smaller individual models into a larger model leading to superior performance compared to the individual models in a diverse range of supervised learning problems [24,25,26,27,28,29], including generalization to out-of-domain (OOD) datasets [30,31,32,33,34]. Knowledge Distillation is a popular technique [35,36,37,38,39,40,41] to transfer the superior performance of the ensemble (teacher) into a single smaller model (student). This technique has been shown to improve the generalization ability of the student models [38,40,42,18].…”

“…Knowledge Distillation is a popular technique [35,36,37,38,39,40,41] to transfer the superior performance of the ensemble (teacher) into a single smaller model (student). This technique has been shown to improve the generalization ability of the student models [38,40,42,18].…”

“…While the Variants Ensemble is consistently more robust and has better clean accuracy than any of the individual variants, it is also computationally more expensive. Knowledge Distillation can be used to compress the knowledge in an ensemble (teacher) into a single model (student) [35,37,38,39,40,41]. Using this technique, we separately trained a ResNet18, a standard VOneResNet18 and a No Noise variant by distilling the Variants Ensemble into these three models (Section B.4).…”

Facial Age Estimation Models for Embedded Systems: A Comparative Study

“…The proposed model is relatively compact and suitable for use on mobile and embedded devices. Greco et al [35] propose an effective training method for age estimation CNNs based on knowledge distillation. The goal of the method is to first learn age estimation using a more complex neural network and then distill the knowledge to a smaller, more compact model.…”