Teacher Guided Neural Architecture Search for Face Recognition

Abstract

Knowledge distillation is an effective tool to compress large pre-trained convolutional neural networks (CNNs) or their ensembles into models applicable to mobile and embedded devices. However, with expected flops or latency, existing methods are hand-crafted heuristics. They propose to pre-define the target student network for knowledge distillation, which may be sub-optimal because it requires much effort to explore a powerful student from the large design space. In this paper, we develop a novel teacher guided neural architecture search method to directly search for a student network with flexible channel and layer sizes. Specifically, we define the search space as the number of the channels/layers, which is sampled based on the probability distribution and is learned by minimizing the search objective of the student network. The maximum probability for the size in each distribution serves as the final searched width and depth of the target student network. Extensive experiments on a variety of face recognition benchmarks have demonstrated the superiority of our method over the state-of-the-art alternatives.

Toward Extremely Lightweight Distracted Driver Recognition With Distillation-Based Neural Architecture Search and Knowledge Transfer

“…With the prior knowledge about typical properties of architectures, NAS approaches commonly define the searching space as a large set of operations (e.g., convolution, fully-connected, and pooling). Each possible architecture in the searching space is evaluated by a certain evaluation strategy [32], [33] and the searching process is controlled by certain searching algorithms, such as reinforcement learning [33], [35], [36], evolutionary search [37], differentiable search [38], or other learning algorithms [34], [39], [40], [41]. NAS commonly defines a searching space at first and then uses a certain policy to generate a sequence of actions in the searching space to specify the architecture.…”

PocketNet: Extreme Lightweight Face Recognition Network Using Neural Architecture Search and Multistep Knowledge Distillation

“…To reduce the NAS search time, all mentioned NAS algorithms proposed to learn from small training datasets such as CIFAR-10 [20] and then utilized the discovered architecture to train on larger datasets such as ImageNet [21]. This advancement in NAS solutions has only recently captured the attention of biometric recognition solutions [22], [23], however, with no deployments towards lightweight or embedded architectures.…”