CMix-NN: Mixed Low-Precision CNN Library for Memory-Constrained Edge Devices

Capotondi, Alessandro; Rusci, Manuele; Fariselli, Marco; Benini, Luca

doi:10.1109/tcsii.2020.2983648

Cited by 102 publications

(80 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Mixed precision can indeed provide a way to reduce the memory footprint of layers that do not need a high precision representation (using 8-bit for weights and activations), while keeping a higher precision (16-bit representation) for layers that need it. The CMix-NN [ 57 ] library already provides an implementation of convolution functions for various data types of configuration (in 2, 4 and 8 bits). To further improve power consumption and memory footprint, binary neural networks can also be considered.…”

Section: Discussionmentioning

confidence: 99%

Quantization and Deployment of Deep Neural Networks on Microcontrollers

Novac

Hacene

Pégatoquet

et al. 2021

Sensors

106

View full text Add to dashboard Cite

Embedding Artificial Intelligence onto low-power devices is a challenging task that has been partly overcome with recent advances in machine learning and hardware design. Presently, deep neural networks can be deployed on embedded targets to perform different tasks such as speech recognition, object detection or Human Activity Recognition. However, there is still room for optimization of deep neural networks onto embedded devices. These optimizations mainly address power consumption, memory and real-time constraints, but also an easier deployment at the edge. Moreover, there is still a need for a better understanding of what can be achieved for different use cases. This work focuses on quantization and deployment of deep neural networks onto low-power 32-bit microcontrollers. The quantization methods, relevant in the context of an embedded execution onto a microcontroller, are first outlined. Then, a new framework for end-to-end deep neural networks training, quantization and deployment is presented. This framework, called MicroAI, is designed as an alternative to existing inference engines (TensorFlow Lite for Microcontrollers and STM32Cube.AI). Our framework can indeed be easily adjusted and/or extended for specific use cases. Execution using single precision 32-bit floating-point as well as fixed-point on 8- and 16 bits integers are supported. The proposed quantization method is evaluated with three different datasets (UCI-HAR, Spoken MNIST and GTSRB). Finally, a comparison study between MicroAI and both existing embedded inference engines is provided in terms of memory and power efficiency. On-device evaluation is done using ARM Cortex-M4F-based microcontrollers (Ambiq Apollo3 and STM32L452RE).

show abstract

Section: Discussionmentioning

confidence: 99%

Quantization and Deployment of Deep Neural Networks on Microcontrollers

Novac

Hacene

Pégatoquet

et al. 2021

Sensors

106

View full text Add to dashboard Cite

show abstract

“…Notice that the accumulator φ linear I NT (w), I NT (x) is still integer-valued, but requires in general more bits than its inputs (i.e., ε φ will be smaller than ε x and ε w ). quant normalizes φ with an affine transformation of parameters κ and λ, then collapses its values, "converting" it to a representation with less bits (i.e., with bigger ε y ) 1 :…”

Section: Background 21 Qnns and Mixed-precision Qnnsmentioning

confidence: 99%

Enabling mixed-precision quantized neural networks in extreme-edge devices

Bruschi

Garofalo

Conti

et al. 2020

Proceedings of the 17th ACM International Conference on Computing Frontiers

View full text Add to dashboard Cite

The deployment of Quantized Neural Networks (QNN) on advanced microcontrollers requires optimized software to exploit digital signal processing (DSP) extensions of modern instruction set architectures (ISA). As such, recent research proposed optimized libraries for QNNs (from 8-bit to 2-bit) such as CMSIS-NN and PULP-NN. This work presents an extension to the PULP-NN library targeting the acceleration of mixed-precision Deep Neural Networks, an emerging paradigm able to significantly shrink the memory footprint of deep neural networks with negligible accuracy loss. The library, composed of 27 kernels, one for each permutation of input feature maps, weights, and output feature maps precision (considering 8-bit, 4-bit and 2-bit), enables efficient inference of QNN on parallel ultra-low-power (PULP) clusters of RISC-V based processors, featuring the RV32IMCXpulpV2 ISA. The proposed solution, benchmarked on an 8-cores GAP-8 PULP cluster, reaches peak performance of 16 MACs/cycle on 8 cores, performing 21× to 25× faster than an STM32H7 (powered by an ARM Cortex M7 processor) with 15× to 21× better energy efficiency.

show abstract

“…We implemented real-world applications, such as smart-web-cam, face-door, and lights-after-dark, at first hand. Real-world applications are implemented under the assumption that the lightweight JavaScript engine and external libraries [5]- [10] run together. Smart-web-cam recognizes a keyword by periodically sampling the sound sensor and the sensor's value into a keyword-spotting model [6], and then reports the camera frame to a server when a keyword is recognized.…”

Section: Evaluation a Experimental Settingsmentioning

confidence: 99%

“…Second, to support abundant functionality, such as DNNbased machine learning [5]- [8] or IoT connectivity standards [9], [10], several external libraries have been used by recent low-end devices. As these libraries use large buffers, such as intermediate feature map buffers [5]- [8] or request message buffers [9], [10], they require a large amount of memory space. Therefore, to use these libraries within the limited memory capacity of the MCUs, the libraries introduce buffer management optimization techniques [7]- [10] or DNN model compression techniques [5]- [8].…”

Section: Introductionmentioning

confidence: 99%

“…As these libraries use large buffers, such as intermediate feature map buffers [5]- [8] or request message buffers [9], [10], they require a large amount of memory space. Therefore, to use these libraries within the limited memory capacity of the MCUs, the libraries introduce buffer management optimization techniques [7]- [10] or DNN model compression techniques [5]- [8]. For example, CMSIS-NN uses a feature map buffer optimization, referred to as partial-im2col, and a model compression technique, referred to as 8-bit quantization, to reduce the peak memory usage to hundreds of KB [7].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Segment-Based Multiple-Base Compressed Addressing for Flexible JavaScript Heap Allocation

Hong

Shin

2020

IEEE Access

View full text Add to dashboard Cite

Many Internet-of-Things (IoT) systems use lightweight JavaScript engines to support easy programming in microcontrollers. Lightweight JavaScript engines use several techniques for memory optimization, such as static heap reservation and compressed addressing. Recent IoT systems also use several external libraries and a larger on-chip memory to support abundant functionalities, such as machine learning and connectivity. However, as the JavaScript heap space is not resizable owing to the memory optimizations, the JavaScript engine or an external library is prone to fail the memory allocation in these devices. To address this problem, we propose a flexible memory optimization technique, segment-based multiple-base compressed addressing (SMBCA), which compresses a pointer indicating a JavaScript object allocated to a resizable heap based on multiple base addresses. SMBCA comprises two components: a dynamic segment allocator (DSA) and a multiple-base compressed address translator (MBCAT). DSA dynamically allocates the JavaScript heap in segment units. Meanwhile, MBCAT converts a low-bitwidth address into a full-bitwidth address and vice versa, based on the multiple base addresses. To reduce the address compression overhead of MBCAT, we propose a software cache technique, reverse map cache (RMC). We found that the SMBCA reduces average memory usage by 43.9% compared to the existing lightweight JavaScript engines when running SunSpider benchmarks, V8 benchmarks, and real-world applications. We also showed that the RMC reduces the average address compression latency of MBCAT by 34.9% when running the SunSpider benchmarks. INDEX TERMS Compressed addressing, Internet of Things, JavaScript, memory management, microcontroller.

show abstract

CMix-NN: Mixed Low-Precision CNN Library for Memory-Constrained Edge Devices

Cited by 102 publications

References 12 publications

Quantization and Deployment of Deep Neural Networks on Microcontrollers

Quantization and Deployment of Deep Neural Networks on Microcontrollers

Enabling mixed-precision quantized neural networks in extreme-edge devices

Segment-Based Multiple-Base Compressed Addressing for Flexible JavaScript Heap Allocation

Contact Info

Product

Resources

About