A Defect Detection Model for Industrial Products Based on Attention and Knowledge Distillation

Abstract

Industrial quality detection is one of the important fields in machine vision. Big data analysis, the Internet of Things, edge computing, and other technologies are widely used in industrial quality detection. Studying an industrial detection algorithm that can be organically combined with the Internet of Things and edge computing is imminent. Deep learning methods in industrial quality detection have been widely proposed recently. However, due to the particularity of industrial scenarios, the existing deep learning-based general object detection methods have shortcomings in industrial applications. This study designs two isomorphic industrial detection models to solve these problems: T-model and S-model. Both proposed models combine swin-transformer with convolution in the backbone and design a residual fusion path. In the neck, this study designs a dual attention module to improve feature fusion. Second, this study presents a knowledge distiller based on the dual attention module to improve the detection accuracy of the lightweight S-model. According to the analysis of the experimental results on four public industrial defect detection datasets, the model in this study is more advantageous in industrial defect detection.

Deep Learning for Automatic Vision-Based Recognition of Industrial Surface Defects: A Survey

“…Considering the shortcomings related to the use of transformers in the industrial defect recognition field, the authors can find different transformer-based approaches applied to the defect detection task. Both Gao et al [283] and Zhang et al [269] proposed a swin-transformer model. The former designed a new window-shift scheme that further strengthened the feature transfer between the windows.…”

“…The pseudo-labels are the output probabilities from the teacher and these concur to guide the student learning process along with the hard (i.e., real) labels. During training, the student model endeavours to match the teacher output probabilities regulated by the shared weights; thus, a ''distillation loss'', sometimes assisted by an attention module [269], steers the probability distribution generated by the student model towards that provided by the teacher model. The loss minimization process can be related to a network-level, whenever the knowledge is optimized only by the last layers or to a channel-level if knowledge is optimized at different levels of feature maps, de facto computing multiple losses at time.…”

AAL-Net: A Lightweight Detection Method for Road Surface Defects Based on Attention and Data Augmentation

“…In the field of industrial surface defect detection, Li et al [39] proposed an attention enhancement-based PANet feature fusion method. Zhang et al [40] proposed a lightweight industrial training model based on knowledge distillation while also incorporating the transformer self-attention used to extract global features. Wang et al [41] explored different types of shallow and lightweight neural networks, including supervised and unsupervised models, to improve the fall detection results.…”