Enabling mixed-precision quantized neural networks in extreme-edge devices

Abstract

The deployment of Quantized Neural Networks (QNN) on advanced microcontrollers requires optimized software to exploit digital signal processing (DSP) extensions of modern instruction set architectures (ISA). As such, recent research proposed optimized libraries for QNNs (from 8-bit to 2-bit) such as CMSIS-NN and PULP-NN. This work presents an extension to the PULP-NN library targeting the acceleration of mixed-precision Deep Neural Networks, an emerging paradigm able to significantly shrink the memory footprint of deep neural networks with negligible accuracy loss. The library, composed of 27 kernels, one for each permutation of input feature maps, weights, and output feature maps precision (considering 8-bit, 4-bit and 2-bit), enables efficient inference of QNN on parallel ultra-low-power (PULP) clusters of RISC-V based processors, featuring the RV32IMCXpulpV2 ISA. The proposed solution, benchmarked on an 8-cores GAP-8 PULP cluster, reaches peak performance of 16 MACs/cycle on 8 cores, performing 21× to 25× faster than an STM32H7 (powered by an ARM Cortex M7 processor) with 15× to 21× better energy efficiency.

Low-precision Logarithmic Number Systems

“…. By selecting the position of the fixed point, standard base-2 LNS can represent implicit bases such as 8 √ 2, 4 √ 2, 2 √ 2, 2, 4, 8, and so on. Remarkably, despite a very extensive literature on LNS over several decades, we have been unable to find a statement of this simple identity in the literature.…”

“…Smaller word sizes can reduce the memory footprint of data, and the complexity of arithmetic for a variety of number systems [9,14,17,33], which will have a significant impact on the ability to deploy systems in resource constrained embedded devices deployed on the edge of networks. Further, fixed-point number systems with very short word lengths have been proposed in the literature for a variety of signal processing applications [2,24], while shorter floating-and fixedpoint numbers have also been used for neural networks [8,18,19,39,43,44].…”

Dustin: A 16-Cores Parallel Ultra-Low-Power Cluster with 2b-to-32b Fully Flexible Bit-Precision and Vector Lockstep Execution Mode

“…However, the extensions proposed in Garofalo et al [10] only tackle part of the challenge, lacking featuring support for mixed-precision operations. Mixed-precision execution requires data conversion and packing/unpacking operations leading to significant overheads if not natively supported by the underlying hardware [11]. When applied to DNNs, exploiting mixed-precision computations on stateof-the-art processors dramatically reduces the memory footprint enabling the execution of MobileNets on tiny endnodes.…”

A Heterogeneous RISC-V Processor for Efficient DNN Application in Smart Sensing System

“…Some researches exploit the property of DNN to reduce latency by using the parallel characteristics of special acceleration circuit design, such as [ 8 , 9 , 10 , 11 , 12 , 13 , 14 ]. Yet these works ignore that the whole power consumption exceeds budget.…”

“…To alleviate the poor-performance problems, a number of studies have been undertaken to accelerate DNN implementations by designing hardware-accelerated intelligent computing architecture for sensing system. Some researches exploit the property of DNN to reduce latency by using the parallel characteristics of special acceleration circuit design, such as [8][9][10][11][12][13][14]. Yet these works ignore that the whole power consumption exceeds budget.…”