CP-NAS: Child-Parent Neural Architecture Search for 1-bit CNNs

Abstract

<jats:p>Neural architecture search (NAS) proves to be among the best approaches for many tasks by generating an application-adaptive neural architectures, which are still challenged by high computational cost and memory consumption. At the same time, 1-bit convolutional neural networks (CNNs) with binarized weights and activations show their potential for resource-limited embedded devices. One natural approach is to use 1-bit CNNs to reduce the computation and memory cost of NAS by taking advantage of the strengths of each in a unified framework. To this end, a Child-Parent model is introduced to a differentiable NAS to search the binarized architecture(Child) under the supervision of a full-precision model (Parent). In the search stage, the Child-Parent model uses an indicator generated by the parent and child model accuracy to evaluate the performance and abandon operations with less potential. In the training stage, a kernel level CP loss is introduced to optimize the binarized network. Extensive experiments demonstrate that the proposed CP-NAS achieves a comparable accuracy with traditional NAS on both the CIFAR and ImageNet databases. It achieves an accuracy of 95.27% on CIFAR-10, 64.3% on ImageNet with binarized weights and activations, and a 30% faster search than prior arts.</jats:p>

Systematic Review on Neural Architecture Search

“…For this purpose, the dataset is divided into training and validation sets to be used for training a model and evaluating its performance, respectively. The results of the VSt can be directly fed into SSp and SSt for their modifications [25,26].…”

“…As a result, the optimized model needed less memory and fewer numbers of Floating Point Operations (FLOPs). CP-NAS (Child-Parent NAS) [25] and DARTS (differentiable architecture search) [42] developed continuous variables for each connection between layers. The optimizer continuously changes the variables using gradient-based methods to find the best suitable connections by emphasizing the important connections with higher values.…”

A Review of Recent Advances of Binary Neural Networks for Edge Computing

“…In binarized neural architecture search (BNAS) [35], neural architecture search is used to search BNNs, and the BNNs obtained by BNAS can outperform conventional models by a large margin. Another natural approach is to use 1-bit CNNs to reduce the computation and memory cost of NAS by taking advantage of the strengths of each in a unified framework [36]. To accomplish this, a Child-Parent (CP) model is introduced to a differentiable NAS to search the binarized architecture (Child) under the supervision of a full precision model (Parent).…”

“…In our BNAS framework, we show that the BNNs obtained by BNAS can outperform conventional models by a large margin. While BNAS only focuses on the kernel binarization, to achieve 1bit CNNs, our CP-NAS [36] advances this work to binarize both the weights and activations. In CP-NAS, a Child-Parent (CP) model is introduced to a differentiable NAS to search the binarized architecture (Child) under the supervision of a full precision model (Parent).…”

Anti-Bandit Neural Architecture Search for Model Defense

“…Following [49,24,46,47,7], we search for computation cells as the building blocks of the final architecture. Different from these approaches, we search for v (v > 2) kinds of cells instead of only normal and reduction cells.…”