Aiming at the existing fabric defect detection algorithms with their suboptimal detection accuracy, high model complexity, difficult deployment to the edge of the device, and insufficient ability to meet the problem of real-time detection of fabric defects, we propose a lightweight fabric defect detection model (LWFDD-YOLO) based on the improved YOLOv8n. First, a generalized efficient layer attention network with selective kernel attention (GELAN_SKA) is proposed to replace the C2f module, and selective kernel attention is added to the module to adjust the weights of the convolution kernel according to the different features of different scales, reduce the use of computational resources, thus improving the model’s detection performance and efficiency. Second, a cascaded group attention (CGA) mechanism is added to provide a different input segmentation for each head to enhance the feature diversity of the input attention heads and improve the model’s computational efficiency. An ultra-lightweight dynamic up-sampling operator (Dy_sample) is introduced, which employs a point-sampling-based approach to reduce the consumption of computational resources and improve model performance. Finally, the YOLOv8l framework is utilized to construct a complex model teacher network and the features learned from the teacher network are transferred to the lightweight network proposed in this paper, thus further improving the model performance of the algorithm. The experimental results show that on the self-built fabric defect dataset, the accuracy, recall, and mean average precision ( mAP) of our algorithm reach 89.4%, 85.2%, and 87.9%, which are 8.9%, 7.2%, and 4.5% higher than the original model, respectively, the number of parameters of the model decreases by 23.4%, the amount of GFLOPs decreases by 25.6%, and the file size is only 9.2 MB. The detection speed can reach 163.4 FPS on GPU. On the AliCloud Tian Chi dataset, the accuracy, recall, and mAP were also improved by 7.4%, 2%, and 4.4%, respectively. The LWFDD-YOLO algorithm proposed in this study can realize real-time detection of fabric defects with relatively obvious improvement in accuracy, and the model requires less memory and is easier to deploy to edge devices, so can be used as a reference for real-time detection of fabric defects.