3PXNet

Abstract

As the adoption of Neural Networks continues to proliferate different classes of applications and systems, edge devices have been left behind. Their strict energy and storage limitations make them unable to cope with the sizes of common network models. While many compression methods such as precision reduction and sparsity have been proposed to alleviate this, they don't go quite far enough. To push size reduction to its absolute limits, we combine binarization with sparsity in Pruned-Permuted-Packed XNOR Networks (3PXNet), which can be efficiently implemented on even the smallest of embedded microcontrollers. 3PXNets can reduce model sizes by up to 38X and reduce runtime by up to 3X compared with already compact conventional binarized implementations with less than 3% accuracy reduction. We have created the first software implementation of sparse-binarized Neural Networks, released as open source library targeting edge devices. Our library is complete with training methodology and model generating scripts, making it easy and fast to deploy.

Big–Little Adaptive Neural Networks on Low-Power Near-Subthreshold Processors

“…Based on XNOR-Net, Ref. [18] constructed a pruned-permuted-packed network that combines binarization with sparsity to push model size reduction to very low limits. On the Nucleo platforms and Raspberry Pi, 3PXNet achieves a reduction in the model size by up to 38× and an improvement in runtime and energy of 25× compared to already compact conventional binarized implementations with a reduction in accuracy of less than 3%.…”

Machine Learning-Based Predictive Model to Improve Cloud Application Performance in Cloud SaaS

“…Dedicated runtime libraries have also been developed, like ARM CMSIS-NN [43], TensorFlow Lite [21], or Microsoft EdgeML [29,41]. Pruning networks to high sparsity levels to save storage and computation has also been extensively explored [45,57].…”

“…Besides storage compression, binarization offers impressive performance improvements by replacing integer multiplication with bitwise XNOR operations [17,35,56]. Due to their efficiency, multiple software [34,50,54,66], and hardware [6,7,9,15,26,37,44,65] implementations of binarized neural networks have been proposed; as well as further optimizations, such as combining binarization with pruning [57], or memoization [48]. However, BNNs come with disadvantages, the most important of which is accuracy degradation [25,27].…”

“…Broad availability, low cost, and ease-of-use would, in turn, make it possible for people to experiment with an increasingly-broader range of applications that improve human-computer interaction, such as audio and visual wake words, context recognition, and user verification [23]. However, achieving it requires unprecedented efforts on co-optimization of algorithms and hardware to make large and computationally complex models usable on devices with very limited memory and processing power [57].…”

“…Multiple techniques have been proposed to address model compression: quantization [1,30,46,57], which uses lower precision numbers for more efficient storage and computation, pruning [19,31,57] which removes inconsequential weights, compressed models [33,36], and optimized software libraries [21,43]. While all the above knobs are readily available to machine learning researchers, it is not obvious how they interact with hardware configurations, given the specific set of constraints, e.g., cost, latency, size, and user experience.…”