Searching toward pareto-optimal device-aware neural architectures

Abstract

Recent breakthroughs in Neural Architectural Search (NAS) have achieved state-of-the-art performance in many tasks such as image classification and language understanding. However, most existing works only optimize for model accuracy and largely ignore other important factors imposed by the underlying hardware and devices, such as latency and energy, when making inference. In this paper, we first introduce the problem of NAS and provide a survey on recent works. Then we deep dive into two recent advancements on extending NAS into multiple-objective frameworks: MONAS [1] and DPP-Net [2] . Both MONAS and DPP-Net are capable of optimizing accuracy and other objectives imposed by devices, searching for neural architectures that can be best deployed on a wide spectrum of devices: from embedded systems and mobile devices to workstations. Experimental results are poised to show that architectures found by MONAS and DPP-Net achieves Pareto optimality w.r.t the given objectives for various devices. Neural Architecture SearchIn this section, we survey the recent literatures on NAS and summarize them into four categories: (a) reinforcementlearning based methods, (b) evolutionary-algorithm based methods, (c) search acceleration, and (d) multi-objective search. Table. 1 provides the overview and comparisons among these literatures. Problem DefinitionGenerally, the problem of neural architecture search can be formulated into two sub-problems: design "Search Space" and "Search Algorithm" [28] .Search Space As its name suggests, search space represents a set of possible neural networks available to be searched over. Usually, a search space has a numerical representation that contains:• Structure of a neural network, such as the depth of a neural net (i.e., the number of hidden layers) and the width of a particular hidden layer.• Configurations, such as operation/connection types, kernel size, the number of filters.

Resource Usage and Performance Trade-offs for Machine Learning Models in Smart Environments

“…By default, both Auto-WEKA and AutoSklearn optimize for only one metric, such as the error rate or accuracy. MONAS and DPP-Net [36], on the other hand, are natural extensions that search and optimize for multiple device-agnostic and device-aware constraints, resulting in gradually better models for all optimization objectives. The outcome of this process are tuples of objective performances where we can select the ones that are Pareto-optimal, that is they are optimal at least in one of the objectives.…”

“…Similar to MONAS, it optimizes device-related (e.g., memory usage) and device-agnostic (e.g., accuracy or model size) objectives. Both approaches were evaluated in [36], showing that both frameworks are effective and are able to achieves Pareto-optimality with respect to the given objectives. While both resource-aware optimization frameworks are closely related to the objectives of our research, an open source implementation for MONAS and DPP-Net was not available for evaluation and adaptation purposes.…”

Optimality Assessment of Memory-Bounded ConvNets Deployed on Resource-Constrained RISC Cores

“…Neural Architecture Search (NAS) [26] is an emerging approach in which pruning and quantization get embedded into a global search where also the topological parameters of the ConvNet, e.g. number of layers, number of filters, connections between layers, etc., take part to the objective function,…”

Efficacy of Topology Scaling for Temperature and Latency Constrained Embedded ConvNets

“…ConvNets are computing-and memory-intensive models that need aggressive compression to fit low-power CPUs. Along with custom compression pipelines based on quantization [4], pruning [5] and neural architecture search [6], a common trend today is to offer end-users a portfolio of pre-trained models with the same back-bone topology but variable size and hence, a different latency-accuracy trade-off [7][8][9][10][11]. One can pick the implementation that best fits the available computing architecture and the application requirements, therefore reducing the design time.…”