Multi-Complexity-Loss DNAS for Energy-Efficient and Memory-Constrained Deep Neural Networks

Enhancing Neural Architecture Search with Multiple Hardware Constraints for Deep Learning Model Deployment on Tiny IoT Devices

“…HardCoRe-NAS [21] combines DNAS with Integer Linear Programming (ILP), using Block Coordinate Stochastic Frank-Wolfe Algorithm to guide the search towards an optimal architecture while remaining in a feasibility region where a latency constraint is satisfied. In our previous work [23] we considered for the first time multiple cost dimensions in a DNAS optimization, where one dimension (model size) is treated as a constraint, and another (number of OPs) as an objective.…”

“…We demonstrate DUCCIO's flexibility by applying it on top two different state-of-the-art DNAS methods, using radically different optimization mechanisms [17], [24]. Note that neither scenario 1) nor scenario 2) were approachable with our preliminary work of [23], which additionally, was tested only on mask-based DNAS. In the rest of the manuscript, we focus on NAS for Convolutional Neural Networks (CNNs) although DUCCIO is also applicable to other types of DNN.…”

“…Therefore, sweeping λ, trying tens of different values at least, is the only way to generate models with the desired cost and accuracy characteristics. Recently, UDC [22], LightNAS [20] and our previous work of [23] added an explicit cost target T in the regularization term, and let the optimizer reduce the difference between the estimated cost and the target, by adding a term proportional to |R(θ) − T |, or equivalent, to the loss function.…”

“…Although making an important step forward, these methods do not solve completely the issues above. First, except for [23] they still consider a single cost dimension, thus not solving issue 1). Similarly, issue 3) is also not eliminated, since once R(θ) becomes < T , the new loss component starts to have an opposite effect, favoring more costly models to reduce the difference from the target.…”

“…In this work, which largely and highly extends [23], we bridge this gap, by proposing a set of objective formulations and training recipes that collectively enable any DNAS, regardless of its specific optimization method, to produce DNNs respecting a complex set of constraints, in a timespan comparable with a single training. We name our method DNAS Under Combined Constraints In One-shot (DUCCIO).…”