To Compress, or Not to Compress: Characterizing Deep Learning Model Compression for Embedded Inference

Abstract

The recent advances in deep neural networks (DNNs) make them attractive for embedded systems. However, it can take a long time for DNNs to make an inference on resourceconstrained computing devices. Model compression techniques can address the computation issue of deep inference on embedded devices. This technique is highly attractive, as it does not rely on specialized hardware, or computation-offloading that is often infeasible due to privacy concerns or high latency. However, it remains unclear how model compression techniques perform across a wide range of DNNs. To design efficient embedded deep learning solutions, we need to understand their behaviors. This work develops a quantitative approach to characterize model compression techniques on a representative embedded deep learning architecture, the NVIDIA Jetson Tx2. We perform extensive experiments by considering 11 influential neural network architectures from the image classification and the natural language processing domains. We experimentally show that how two mainstream compression techniques, data quantization and pruning, perform on these network architectures and the implications of compression techniques to the model storage size, inference time, energy consumption and performance metrics. We demonstrate that there are opportunities to achieve fast deep inference on embedded systems, but one must carefully choose the compression settings. Our results provide insights on when and how to apply model compression techniques and guidelines for designing efficient embedded deep learning systems.

Adaptive Computation Offloading for Mobile Augmented Reality

“…Deep learning techniques have shown astonishing success in various tasks, including object detection [21,31,71,76]. Efforts have been devoted to accelerating the inference of deep learning models on mobile devices through model compression techniques [17,36,56,69,76]. Actor leverages the recent advances in deep inference acceleration to speed up the DNN model execution time on the remote server to reduce the latency.…”

“…Existing work on computation offloading for mobile computer vision has considered a range of techniques. These include trading DNN accuracy for lower latency [48,69], offloading the entire [16,45] or part of the object detection task [52,119] to the cloud. Although these approaches were effective for the traditional high definition (HD) video format with a resolution of up to 1,920 × 1,080, they would require over 300ms offloading latency even using a high-speed 5G network when processing a 4K video, and are thus ill-suited for emerging 4K MAR.…”

A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions

“…Further, we categorized the DNN compression using sparse representation into three sub-categories, i.e., quantization, multiplexing, and weight sharing. Section IVpresents a detailed description of the sparse representation techniques in the existing literature [35], [37], [38], [43], [45], [48]- [52] on DNN compression.…”

“…Channel pruning [25]- [29], [83] Filter pruning [30]- [34] Connection pruning [35]- [43], [83] Layer pruning [43]- [45] Sparse representation Quantization [35], [38], [45], [48] Multiplexing [37], [49], [50] Weight sharing [35], [37], [43], [51], [52] Bits precision Estimation using integer [48], [54] Low bits representation [55] Binarization [41], [56]- [58] Knowledge distillation…”

“…Authors in [38] developed a quantitative characterization approach for a deep learning model to make its deployment on embedded devices. It not only provides a better selection mechanism for deployment of DNN model on the embedded device but also guides the development of compression technique using a wiser connection pruning.…”