Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression

Blakeney, Cody; Li, Xiaomin; Yan, Yan; Zong, Ziliang

doi:10.1109/tpds.2020.3047003

Cited by 28 publications

(16 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Contrary to the earlier belief that teachers must be larger than students, recent studies have revealed that smaller teachers can be used to train larger students [10]. With such findings, blockwise distillation is used in various fields such as model compression [7,11] and NAS [9,12]. Since training a small teacher for a new task is quick and easy, blockwise distillation can be applied in most cases where traditional training is used.…”

Section: Introductionmentioning

confidence: 98%

See 1 more Smart Citation

Pipe-BD: Pipelined Parallel Blockwise Distillation

Hongsun¹,

Jung²,

Song³

et al. 2023

Preprint

View full text Add to dashboard Cite

Training large deep neural network models is highly challenging due to their tremendous computational and memory requirements. Blockwise distillation provides one promising method towards faster convergence by splitting a large model into multiple smaller models. In state-of-the-art blockwise distillation methods, training is performed block-by-block in a data-parallel manner using multiple GPUs. To produce inputs for the student blocks, the teacher model is executed from the beginning until the current block under training. However, this results in a high overhead of redundant teacher execution, low GPU utilization, and extra data loading. To address these problems, we propose Pipe-BD, a novel parallelization method for blockwise distillation. Pipe-BD aggressively utilizes pipeline parallelism for blockwise distillation, eliminating redundant teacher block execution and increasing per-device batch size for better resource utilization. We also extend to hybrid parallelism for efficient workload balancing. As a result, Pipe-BD achieves significant acceleration without modifying the mathematical formulation of blockwise distillation. We implement Pipe-BD on PyTorch, and experiments reveal that Pipe-BD is effective on multiple scenarios, models, and datasets.

show abstract

Section: Introductionmentioning

confidence: 98%

“…Blockwise distillation [7,8,9] is one promising approach to mitigate such problems. As illustrated in Fig.…”

Section: Introductionmentioning

confidence: 99%

Pipe-BD: Pipelined Parallel Blockwise Distillation

Hongsun¹,

Jung²,

Song³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In summary, the significant contributions of this study are as follows: Instead of designing complex GCN structures, we introduce the concept of knowledge distillation into GCN‐based recommendations and propose a TKDM to improve the recommendation effectiveness. Specifically, we design new knowledge distillation methods for GCN‐based recommendation systems by noting that existing ones 30–32 are proposed for classification tasks on traditional neural network components, for example, MLP and convolutional neural networks. We present and apply a SDM on an FRL‐net that effectively learns user and item feature representations. We then introduce a MDM and apply it on a UPL‐net to learn user preferences on items accurately. We comprehensively investigate the effectiveness of the proposed TKDM through extensive experiments on three real‐world data sets.…”

Section: Introductionmentioning

confidence: 99%

“…(1) Instead of designing complex GCN structures, we introduce the concept of knowledge distillation into GCN-based recommendations and propose a TKDM to improve the recommendation effectiveness. Specifically, we design new knowledge distillation methods for GCN-based recommendation systems by noting that existing ones [30][31][32] are proposed for classification tasks on traditional neural network components, for example, MLP and convolutional neural networks. (2) We present and apply a SDM on an FRL-net that effectively learns user and item feature representations.…”

Section: Introductionmentioning

confidence: 99%

A two‐phase knowledge distillation model for graph convolutional network‐based recommendation

Huang

Lin

Gong

et al. 2022

Int J of Intelligent Sys

View full text Add to dashboard Cite

Graph convolutional network (GCN)‐based recommendation has recently attracted significant attention in the recommender system community. Although current studies propose various GCNs to improve recommendation performance, existing methods suffer from two main limitations. First, user–item interaction data is generally sparse in practice, highlighting these methods' ineffectiveness in learning user and item feature representations. Second, they usually perform a dot‐product operation to model and calculate user preferences on items, leading to inaccurate user preference learning. To address these limitations, this study adopts a design idea that sharply differs from existing works. Specifically, we introduce the knowledge distillation concept into GCN‐based recommendation and propose a two‐phase knowledge distillation model (TKDM) improving recommendation performance. In Phase I, a self‐distillation method on a graph auto‐encoder learns the user and item feature representations. This auto‐encoder employs a simple two‐layer GCN as an encoder and a fully connected layer as a decoder. On this basis, in Phase II, a mutual‐distillation method on a fully connected layer is introduced to learn user preferences on items with triple‐based Bayesian personalized ranking. Extensive experiments on three real‐world data sets demonstrate that TKDM outperforms classic and state‐of‐the‐art methods related to GCN‐based recommendation problems.

show abstract

“…DNN compression methods reduce the model's memory size and computational requirements without significantly impacting its accuracy. DNN compression techniques are classified into five main categories [6]: pruning [7], quantization [8], compact convolutional filters [9], knowledge distillation [10], and low-rank factorization [11].…”

Section: Introductionmentioning

confidence: 99%

A Design Space Exploration Methodology for Enabling Tensor Train Decomposition in Edge Devices

Kokhazadeh

Κεραμίδας

Kelefouras

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Deep Neural Networks (DNN) have made significant advances in various fields, including speech recognition and image processing. Typically, modern DNNs are both compute and memory intensive and as a consequence their deployment on edge devices is a challenging problem. A well-known technique to address this issue is Low-Rank Factorization (LRF), where a weight tensor is approximated with one or more lower-rank tensors, reducing the number of executed instructions and memory footprint. However, finding an efficient solution is a complex and time-consuming process as LRF includes a huge design space and different solutions provide different trade-offs in terms of FLOPs, memory size, and prediction accuracy. In this work a methodology is presented that formulates the LRF problem as a (FLOPs vs. memory vs. prediction accuracy) Design Space Exploration (DSE) problem. Then the DSE space is drastically pruned by removing inefficient solutions. Our experimental results prove that it is possible to output a limited set of solutions with better accuracy, memory, and FLOPs compared to the original (non-factorized) model. Our methodology has been developed as a standalone, parameterized module integrated into T3F library of TensorFlow 2.X.

show abstract

Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression

Cited by 28 publications

References 18 publications

Pipe-BD: Pipelined Parallel Blockwise Distillation

Pipe-BD: Pipelined Parallel Blockwise Distillation

A two‐phase knowledge distillation model for graph convolutional network‐based recommendation

A Design Space Exploration Methodology for Enabling Tensor Train Decomposition in Edge Devices

Contact Info

Product

Resources

About