Head Network Distillation: Splitting Distilled Deep Neural Networks for Resource-Constrained Edge Computing Systems

Abstract

As the complexity of Deep Neural Network (DNN) models increases, their deployment on mobile devices becomes increasingly challenging, especially in complex vision tasks such as image classification. Many of recent contributions aim either to produce compact models matching the limited computing capabilities of mobile devices or to offload the execution of such burdensome models to a compute-capable device at the network edge-the edge servers. In this paper, we propose to modify the structure and training process of DNN models for complex image classification tasks to achieve in-network compression in the early network layers. Our training process stems from knowledge distillation, a technique that has been traditionally used to build small-student-models mimicking the output of larger-teacher-models. Here, we adopt this idea to obtain aggressive compression while preserving accuracy. Our results demonstrate that our approach is effective for state-of-the-art models trained over complex datasets, and can extend the parameter region in which edge computing is a viable and advantageous option. Additionally, we demonstrate that in many settings of practical interest we reduce the inference time with respect to specialized models such as MobileNet v2 executed at the mobile device, while improving accuracy.

Communication-Oriented Model Fine-Tuning for Packet-Loss Resilient Distributed Inference Under Highly Lossy IoT Networks

“…Unlike the research on vertical DI [26], [27], we have focused on the horizontal DI, because the sharing of the raw input in the vertical DI includes a critical privacy risk. In the horizontal DI literature, some works have addressed the achievement of low communication latency by optimizing the division point [5], [14], [15], leveraging multiple sink nodes [13], pruning the DNN model [14], quantizing the message [15]- [17], dimensional reduction of the message [18]- [20], and combining multiple inference tasks into a single one [16]. However, these works assumed a reliable communication link and aimed to reduce the communication payload size.…”

BottleFit: Learning Compressed Representations in Deep Neural Networks for Effective and Efficient Split Computing

“…Most of the existing studies train the altered models from scratch [16], [18], [19]. Others reuse pretrained parameters in available architectures for the tail model, while re-designing and retraining the head portion to introduce a bottleneck [17], [21], [22]. These latter contributions introduce the notion of Head Network Distillation (HND) and Generalized HND (GHND), that use knowledge distillation in the training process.…”

“…• We apply BottleFit on cutting-edge CNNs such as DenseNet-169, DenseNet-201 and ResNet-152 on the ImageNet dataset, and compare the accuracy obtained by BottleFit with state-of-the-art local computing [6] and split computing approaches [16]- [19], [21], [24]. Our training campaign concludes that BottleFit achieves up to 77.1% data compression (with respect to JPEG) with only up to 0.6% loss in accuracy, while existing mobile and split computing approaches incur considerable accuracy loss of up to 6% and 3.6%, respectively.…”

Low Latency Deep Learning Inference Model for Distributed Intelligent IoT Edge Clusters

“…create an ecosystem for achieving inference models with excellent performance. In this regard, many attempts have been reported to optimize the DNN models at edge devices [61], [62]. While Communication load, communication overhead, cost, memory, processing speed, network bandwidth, jitter, complexity are a few performance parameters, much of the preliminary research has focused on low-latency and energyefficient computations.…”