EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search

Abstract

Neural Architecture Search (NAS) has shown excellent results in designing architectures for computer vision problems. NAS alleviates the need for human-defined settings by automating architecture design and engineering. However, NAS methods tend to be slow, as they require large amounts of GPU computation. This bottleneck is mainly due to the performance estimation strategy, which requires the evaluation of the generated architectures, mainly by training them, to update the sampler method. In this paper, we propose EPE-NAS, an efficient performance estimation strategy, that mitigates the problem of evaluating networks, by scoring untrained networks and creating a correlation with their trained performance. We perform this process by looking at intra and inter-class correlations of an untrained network. We show that EPE-NAS can produce a robust correlation and that by incorporating it into a simple random sampling strategy, we are able to search for competitive networks, without requiring any training, in a matter of seconds using a single GPU. Moreover, EPE-NAS is agnostic to the search method, since it focuses on the evaluation of untrained networks, making it easy to integrate into almost any NAS method.

ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients

“…In this section, we further provide the comparison between our proposed ZiCo and more proxies proposed recently: KNAS ), NASWOT (Lopes et al (2021)), GradSign ( Zhang & Jia (2022)), and NTK (TE-NAS Chen et al (2021b), NASI Shu et al (2022a)). To compute the correlations, we use the official code released by the authors of the above papers to obtain the values Table 4: The correlation coefficients between various zero-cost proxies and two naive proxies (#Params and FLOPs) vs. test accuracy on NATSBench-SSS and NATSBench-TSS (KT and SPR represent Kendall's τ and Spearman's ρ, respectively).…”

“…Moreover, the Zen-score approximates the gradient w.r.t featuremaps and measures the complexity of neural networks ; . Furthermore, Jacob cov leverages the Jacobian matrix between the loss and multiple input samples to quantify the capacity of modeling the complex functions Lopes et al (2021).…”

Towards Less Constrained Macro-Neural Architecture Search

“…Model-based predictors predict the final validation accuracy of an architecture based on its encodings [23,30,61]. Zero-cost proxies look at architectures at initialization stage, and calculate statistics that correlate with the architecture's final validation accuracy [6,39,45]. White et.…”

“…The validation accuracy is used as an indication if the architecture is capable of learning from the small number of examples shown, which ultimately can be used to distinguish between architectures that can be trained efficiently from those that can't. The proposed method then looks at the capability of the untrained architecture at initialization stage to model complex functions, through Jacobian analysis [39,45]. For this, one can define a mapping from the input x i ∈ R D , through the net-work, w(x i ), where x i represents an image that belongs to a batch X, and D is the input dimension.…”

Improving Neural Architecture Search by Mixing a FireFly algorithm with a Training Free Evaluation

“…One of the main problem is that training a model to evaluate its performance is time consuming, resulting in a huge computation time, which can take days even using hundred of GPUs [22]. Therefore methods exists to bypass this learning phase and evaluate an architecture from metrics based on the distribution of cells activation relative to different inputs values gather in a mini-batch, such as Mellor's metric [22] and Lopes et al [23] proposal.…”