HW-FlowQ: A Multi-Abstraction Level HW-CNN Co-design Quantization Methodology

Abstract

Model compression through quantization is commonly applied to convolutional neural networks (CNNs) deployed on compute and memory-constrained embedded platforms. Different layers of the CNN can have varying degrees of numerical precision for both weights and activations, resulting in a large search space. Together with the hardware (HW) design space, the challenge of finding the globally optimal HW-CNN combination for a given application becomes daunting. To this end, we propose HW-FlowQ, a systematic approach that enables the co-design of the target hardware platform and the compressed CNN model through quantization. The search space is viewed at three levels of abstraction, allowing for an iterative approach for narrowing down the solution space before reaching a high-fidelity CNN hardware modeling tool, capable of capturing the effects of mixed-precision quantization strategies on different hardware architectures (processing unit counts, memory levels, cost models, dataflows) and two types of computation engines (bit-parallel vectorized, bit-serial). To combine both worlds, a multi-objective non-dominated sorting genetic algorithm (NSGA-II) is leveraged to establish a Pareto-optimal set of quantization strategies for the target HW-metrics at each abstraction level. HW-FlowQ detects optima in a discrete search space and maximizes the task-related accuracy of the underlying CNN while minimizing hardware-related costs. The Pareto-front approach keeps the design space open to a range of non-dominated solutions before refining the design to a more detailed level of abstraction. With equivalent prediction accuracy, we improve the energy and latency by 20% and 45% respectively for ResNet56 compared to existing mixed-precision search methods.

MOHAQ: Multi-Objective Hardware-Aware Quantization of recurrent neural networks

“…EMOMC [48] HW_FlowQ [49] APQ [47] This We have shown that the SRU-based model can be compressed using post-training quantization up to 8x without any error increase and up to 12x with a 1.2 percentage point (p.p.) increase in the error.…”

“…EMOMC (Evolutionary Multi-Objective Model Compression) [48], and HW_FlowQ [49] are methods that apply multi-objective evolutionary optimization to search for the NN model compression parameters. They have different approaches to overcoming the difficulty of applying retraining during the search.…”

“…However, no retraining has been applied to quantized solutions. HW_FlowQ is a co-design method for the CNN model quantization configuration and target hardware platform design [49]. They accounted for training after quantization by relying on an accuracy estimation approach.…”

“…However, the increase they cause in the speedup is negligible. The effect of retraining has been considered using different approaches in the evolutionary-based methods discussed in Section 3 (EMOMC [48], HW_FlowQ [49], and APQ [47]). In Table 9, we compare the state-of-the-art methods against our beacon-based method.…”

“…Thus, their approach is to reduce the search space size. In HW_FlowQ, the training required to compensate for the accuracy loss is divided into two phases [49]. The first phase is during the search.…”