Neural Architecture Search Survey: A Hardware Perspective

Abstract

We review the problem of automating hardware-aware architectural design process of Deep Neural Networks (DNNs). The field of Convolutional Neural Network (CNN) algorithm design has led to advancements in many fields such as computer vision, virtual reality, and autonomous driving. The end-to-end design process of a CNN is a challenging and time-consuming task as it requires expertise in multiple areas such as signal and image processing, neural networks, and optimization. At the same time, several hardware platforms, general- and special-purpose, have equally contributed to the training and deployment of these complex networks in a different setting. Hardware-Aware Neural Architecture Search (HW-NAS) automates the architectural design process of DNNs to alleviate human effort, and generate efficient models accomplishing acceptable accuracy-performance tradeoffs. The goal of this paper is to provide insights and understanding of HW-NAS techniques for various hardware platforms (MCU, CPU, GPU, ASIC, FPGA, ReRAM, DSP, and VPU), followed by the co-search methodologies of neural algorithm and hardware accelerator specifications.

Dynamic nsNET2: Efficient Deep Noise Suppression with Early Exiting

“…It consists of exploring the search space of possible neural network architectures to optimize them according to one or several metrics, such as model accuracy, size, or computational complexity. Within NAS, several methods for traversing the vast search space of possible architectures have been proposed, including gradient-based methods [27], Reinforcement Learning [5], and Genetic Algorithms (GA). In particular, the latter has been adopted in computer vision as a way to find an optimal CNN structure for face recognition [24] and, more recently, achieving better performances than manuallyderived CNNs for image classification [23].…”

HaeNAS: Hardware-Aware Efficient Neural Architecture Search via Zero-Cost Proxy

“…Recently, pruning-at-initialization methods [4,5] have attracted a lot of interest as they can reduce the training cost while achieve similar accuracy. Neural Architecture Search (NAS) has become a popular approach for automatically discovering competitive neural network architectures, surpassing those designed by humans [2]. Existing methods [1,12] primarily focus on accuracy while neglecting other hardware-related factors, such as latency, energy, and memory.…”

TA-DARTS: Temperature Annealing of Discrete Operator Distribution for Effective Differential Architecture Search

“…With the rapid advancement of artificial intelligence (AI) technologies, the demand for constructing AI models has surged [1][2][3][4][5]. Concurrently, the endeavor of devising neural architectures within machine learning has emerged as a time-intensive and laborious task, mainly due to the inherent challenge of attaining optimal neural architectures solely through expert insights [1,5,6]. More precisely, the pursuit of optimal or sufficiently effective architectures frequently entails considerable time and effort.…”

“…More precisely, the pursuit of optimal or sufficiently effective architectures frequently entails considerable time and effort. The complexity stems from the inherent diversity of plausible neural architectures for a given task, necessitating recourse to a brute-force methodology involving training for each architecture to pinpoint the optimal configuration [1][2][3]5,6]. This backdrop has spurred a surge in AutoML research, particularly within the domain of Neural Architecture Search (NAS), which strives to streamline the exploration of efficient architectures [4,6].…”