Lightweight Tunnel Defect Detection Algorithm Based on Knowledge Distillation

Жу, Анфу; Wang, Bin; Xie, Jiaxiao; Ma, Congxiao

doi:10.3390/electronics12153222

Cited by 3 publications

(2 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Researchers have proposed a range of CNN compression and acceleration techniques, which include knowledge distillation [1,2], neural network architecture search [3,4], pruning [5,6], and quantization [7]. Knowledge distillation uses a large model as a 'teacher' to guide the training of a smaller 'student' model, enabling the smaller model to assimilate the knowledge contained in the larger model.…”

Section: Introductionmentioning

confidence: 99%

“…Knowledge distillation uses a large model as a 'teacher' to guide the training of a smaller 'student' model, enabling the smaller model to assimilate the knowledge contained in the larger model. For example, Anfu Zhu [1] achieved an 81.4% reduction in the number of parameters of the student model compared to the teacher model by utilizing multi-dimensional knowledge distillation, increasing accuracy by 2.5% over training the student model directly. Neural architecture search is a technique that employs automated methods to discover optimal neural network structures, effectively balancing network accuracy and efficiency.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

AE-Qdrop: Towards Accurate and Efficient Low-Bit Post-Training Quantization for A Convolutional Neural Network

Li,

Chen,

Jin

et al. 2024

Electronics

View full text Add to dashboard Cite

Blockwise reconstruction with adaptive rounding helps achieve acceptable 4-bit post-training quantization accuracy. However, adaptive rounding is time intensive, and the optimization space of weight elements is constrained to a binary set, thus limiting the performance of quantized models. The optimality of block-wise reconstruction requires that subsequent network blocks remain unquantized. To address this, we propose a two-stage post-training quantization scheme, AE-Qdrop, encompassing block-wise reconstruction and global fine-tuning. In the block-wise reconstruction stage, a progressive optimization strategy is introduced as a replacement for adaptive rounding, enhancing both quantization accuracy and efficiency. Additionally, the integration of randomly weighted quantized activation helps mitigate the risk of overfitting. In the global fine-tuning stage, the weights of each quantized network block are corrected simultaneously through logit matching and feature matching. Experiments in image classification and object detection tasks validate that AE-Qdrop achieves high precision and efficient quantization. For the 2-bit MobileNetV2, AE-Qdrop outperforms Qdrop in quantization accuracy by 6.26%, and its quantization efficiency is fivefold higher.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

AE-Qdrop: Towards Accurate and Efficient Low-Bit Post-Training Quantization for A Convolutional Neural Network

Li,

Chen,

Jin

et al. 2024

Electronics

View full text Add to dashboard Cite

show abstract

YOLOv7-TID: A Lightweight Network for PCB Intelligent Detection

Zhuo,

Shi,

Zhou

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Automatic detection of structural defects in tunnel lining via network pruning and knowledge distillation in YOLO

Liu,

Zeng

2024

Structural Health Monitoring

View full text Add to dashboard Cite

Cracks in tunnel linings might cause a lack of water tightness, directly affecting the overall stability and durability using an increased risk of corrosion of the rebar. Hence, automatic, timely and accurate detection of cracks is significant to safe operation and maintenance for tunnels. In recent years, Convolution Neural Network (CNN) has achieved success in the field of computer vision. However, the large storage requirement and high computational consumption of the model limit its application. Aiming to overcome these challenges, this study proposes a lightweight target detector YOLOV5-lite, which utilizes transfer learning to construct the YOLO single-stage target detection model and uses the Efficient Intersection over Union ( EIoU) in the loss function to optimize its convergence speed. To improve the model efficiency, a Network Pruning algorithm is performed in order to reduce the number of parameters in the model. To compensate for the loss of accuracy caused by the Network Pruning algorithm compressing the network, knowledge distillation algorithm is implemented in this study and fused with the Network Pruning algorithm. This results in a new lightweight modelling framework which has a high computational efficiency enabling deployment in mobile devices as well as high accuracy leading to good detection performance. To illustrate the advantages of the proposed method, two experiments using an extensive evaluation of the YOLOV5 series of models and a comparison with different model tests were made to validate it. In the evaluation of the YOLOv5 series, key findings include: (a) The optimized YOLOV5-loss model, incorporating the EIoU loss function, achieved an impressive crack recognition accuracy of 0.97. This model demonstrated superior capability in detecting fine cracks, particularly in corner regions, with an accuracy exceeding 0.85. The EIoU loss function offers enhanced sensitivity to overlapping regions and more precise boundary localization, which are critical in identifying minute or boundary-ambiguous cracks. (b) The YOLOV5-finetuned model, which underwent network pruning alone, achieved an accuracy of 0.74 but was hindered by significant detection gaps, despite achieving a 50.9% reduction in model size. In contrast, the YOLOV5-lite model, refined using a combination of network pruning and knowledge distillation, maintained a high recognition accuracy of 0.96, with only a negligible 0.01 difference from the optimal YOLOV5-loss model. When compared with five different models, the YOLOv5-lite model demonstrated significant advantages: A substantial reduction in the size of the proposed YOLOV5-lite by 184.1 MB, 659.4 MB and an increase in the number of the transmitted Frames Per Second (FPS) by 6.69 f/s, 13.24 f/s compared with that of YOLOV3, Faster R-CNN, respectively. Overall, the proposed new method consistently achieves high detection performance as well as substantially reducing computational demands, making it well-suited for real-time applications, particularly in mobile and embedded systems.

show abstract

Lightweight Tunnel Defect Detection Algorithm Based on Knowledge Distillation

Cited by 3 publications

References 28 publications

AE-Qdrop: Towards Accurate and Efficient Low-Bit Post-Training Quantization for A Convolutional Neural Network

AE-Qdrop: Towards Accurate and Efficient Low-Bit Post-Training Quantization for A Convolutional Neural Network

YOLOv7-TID: A Lightweight Network for PCB Intelligent Detection

Automatic detection of structural defects in tunnel lining via network pruning and knowledge distillation in YOLO

Contact Info

Product

Resources

About