Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective

Abstract

Neural Architecture Search (NAS) has been explosively studied to automate the discovery of top-performer neural networks. Current works require heavy training of supernet or intensive architecture evaluations, thus suffering from heavy resource consumption and often incurring search bias due to truncated training or approximations. Can we select the best neural architectures without involving any training and eliminate a drastic portion of the search cost? We provide an affirmative answer, by proposing a novel framework called training-free neural architecture search (TE-NAS). TE-NAS ranks architectures by analyzing the spectrum of the neural tangent kernel (NTK) and the number of linear regions in the input space. Both are motivated by recent theory advances in deep networks and can be computed without any training and any label. We show that: (1) these two measurements imply the trainability and expressivity of a neural network; (2) they strongly correlate with the network's test accuracy. Further on, we design a pruning-based NAS mechanism to achieve a more flexible and superior trade-off between the trainability and expressivity during the search. In NAS-Bench-201 and DARTS search spaces, TE-NAS completes high-quality search but only costs 0.5 and 4 GPU hours with one 1080Ti on CIFAR-10 and ImageNet, respectively. We hope our work inspires more attempts in bridging the theoretical findings of deep networks and practical impacts in real NAS applications. Code is available at: https://github.com/VITA-Group/TENAS.

Neural Architecture Search for Spiking Neural Networks

“…In very recent works, the key focus has been the efficiency of the NAS technique [1,91,92,94,105] owing to the growing size of dataset and architecture. Interestingly, a line of work suggests the concept of NAS without training where the networks do not require training during the search stage [12,58,88]. This can significantly reduce the computational cost for searching optimal architecture.…”

“…This makes it difficult to search for an optimal SNN architecture with NAS techniques that train the architecture candidate multiple times [2,78,[107][108][109] or train a complex supernet [8,29,55,86]. To minimize the training budget, our work is motivated by the previous works [12,58,88] which demonstrate that the optimal architecture can be founded without any training process. Specifically, Mellor et al [58] provide the interesting observation that the architecture having distinctive representations across different data samples is likely to achieve higher posttraining performance.…”

“…NAS without training approaches in ANN domain [12,58] are based on the theoretical concept of linear region in neural networks [32,33,59,67,87]. That is, each piecewise linear function (such as, ReLU) divides the input space into multiple linear regions.…”

“…As SNNs have a significantly slower training process compared to ANNs (e.g., training SNN on MNIST with NVIDIA V100 GPU takes 11.43× more latency compared to the same ANN architecture [53]), the above NAS approaches are difficult to be applied on SNNs. On the other hand, recent works [12,58,88] have proposed efficient NAS approaches that search the best neuron cell from initialized networks without any training. Specifically, [58] shows that the network architecture with a high representation power at initialization is likely to achieve higher post-training accuracy.…”

“…For the first time, we showcase a NAS technique for finding better SNN architecture on the image recognition task. (ii) Motivated by the prior work [12,58,88], we find an SNN-friendly architecture by comparing temporal activation without any training process. Eliminating the training cost to find the optimal architecture brings a huge advantage for SNNs that require significantly longer training time compared to ANNs.…”