UNAS: Differentiable Architecture Search Meets Reinforcement Learning

Abstract

Neural architecture search (NAS) aims to discover network architectures with desired properties such as high accuracy or low latency. Recently, differentiable NAS (DNAS) has demonstrated promising results while maintaining a search cost orders of magnitude lower than reinforcement learning (RL) based NAS. However, DNAS models can only optimize differentiable loss functions in search, and they require an accurate differentiable approximation of non-differentiable criteria. In this work, we present UNAS, a unified framework for NAS, that encapsulates recent DNAS and RL-based approaches under one framework. Our framework brings the best of both worlds, and it enables us to search for architectures with both differentiable and non-differentiable criteria in one unified framework while maintaining a low search cost. Further, we introduce a new objective function for search based on the generalization gap that prevents the selection of architectures prone to overfitting. We present extensive experiments on the CIFAR-10, CIFAR-100 and ImageNet datasets and we perform search in two fundamentally different search spaces. We show that UNAS obtains the state-of-the-art average accuracy on all three datasets when compared to the architectures searched in the DARTS [18] space. Moreover, we show that UNAS can find an efficient and accurate architecture in the ProxylessNAS [28] search space, that outperforms existing MobileNetV2 [28] based architectures.

Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training

“…For example, while EfficientNets vastly outperform ResNets in terms of theoretical training efficiency, they have often been found to underperform when considering practical training efficiency on GPUs (Lee et al, 2020). Some recent work has used NAS to optimise practical efficiency on GPUs (Cai et al, 2018;Vahdat et al, 2019;Lin et al, 2020). For the presented work, we prioritised hang-engineered solutions, but do not rule out NAS methods in future work.…”