ENOS: Energy-Aware Network Operator Search in Deep Neural Networks

Abstract

This work proposes a novel Energy-aware Network Operator Search (ENOS) approach to address the energy-accuracy trade-offs of a deep neural network (DNN) accelerator. In recent years, novel hardwarefriendly inference operators such as binary-weight, multiplication-free, and deep-shift have been proposed to improve the computational efficiency of a DNN accelerator. However, simplifying DNN operators invariably comes at lower accuracy, especially on complex processing tasks. While prior works generally implement the same inference operator throughout the neural architecture, the proposed ENOS framework allows an optimal layer-wise integration of inference operators with optimal precision to maintain high prediction accuracy and high energy efficiency. The search in ENOS is formulated as a continuous optimization problem, solvable using gradient descent methods, thereby minimally increasing the training cost when learning both layer-wise inference operators and weights. Utilizing ENOS, we discuss multiply-accumulate (MAC) cores for digital spatial architectures that can be reconfigured to different operators and varying computing precision. ENOS training methods with single and bi-level optimization objectives are discussed and compared. We also discuss a sequential operator assignment strategy in ENOS that only learns the assignment for one layer in one training step. Furthermore, a stochastic mode of ENOS is also presented. ENOS is characterized on ShuffleNet and SqueezeNet using CIFAR10 and CIFAR100. Compared to the conventional uni-operator approaches, under the same energy budget, ENOS improves accuracy by 10-20%. ENOS also outperforms the accuracy of comparable mixed-precision uni-operator implementations by 3-5% for the same energy budget.INDEX TERMS Low power, deep neural network, mixed-precision learning.

Maintaining Symmetry between Convolutional Neural Network Accuracy and Performance on an Edge TPU with a Focus on Transfer Learning Adjustments

“…Finally, a large amount of prior work has been done on building better edge processors [44][45][46][47][48][49][50][51][52]. Our work does not aim to inform the design of new hardware for machine learning but rather to provide insights and an evaluation methodology for developing machine learning models for edge tensor processors.…”

ADC/DAC-Free Analog Acceleration of Deep Neural Networks With Frequency Transformation

“…For instance, analog computations simplify processing cells and enable the integration of storage and computations within a single cell in many compute-in-memory designs. This significantly reduces data movement during deep learning computations, which is a critical bottleneck for energy and performance in traditional processors [31], [32], [33], [34].…”

A CNN-LSTM and Attention-Mechanism-Based Resistance Spot Welding Quality Online Detection Method for Automotive Bodies

“…This Section proposes an online quality detection model for RSW by combining the CNN-LSTM network and attention mechanism, as shown in Figure 3. The model first uses the CNN-LSTM network to automatically extract local detail features and time-series correlation features of the welding data, and then uses the attention mechanism to focus on the critical main features that are most useful for the output results, improving the model's effectiveness and endowing the model with interpretability [33,34]. The proposed model consists of five parts: an input layer, convolutional neural network (CNN) layer, long short-term memory network (LSTM) layer, attention mechanism layer and output layer.…”