Aiming at the problems of low detection accuracy and high leakage rate in traditional detection algorithms, an improved YOLOv8 algorithm is proposed for automatic detection of fabric defects. A swin transformer block was added to the C2f module in the backbone network, which can transfer information between multiple attention layers in parallel to capture fabric defect information and improve the detection accuracy of small-sized defects. To enhance the model’s performance in detecting defects of various sizes, a bidirectional feature pyramid network (BiFPN) was incorporated into the neck. This allows for the assignment of different weights to defect features in different layers. A convolution block attention module (CBAM) was added to the feature fusion layer, enabling the model to automatically increase the weight of essential features and suppress nonessential features during training to solve the problem of leakage detection of small-sized defects due to occlusion and background confusion. The Wise-IoU (WIoU) loss function replaces the conventional loss function, addressing sample imbalance and directing the model to prioritize average-quality samples. This modification contributes to an overall improvement in the model’s performance. The results of the experiment proved that on the self-constructed fabric defect dataset, the algorithm in this paper achieved an accuracy of 97.7%, recall of 95.1%, and mAP of 96.8%, which are 4.4%, 9.4%, and 5.1% higher than those of the YOLOv8 algorithm, respectively. On the AliCloud Tianchi dataset, the algorithm achieves 52.3%, 49.2%, and 49.8% in terms of accuracy, recall, and mAP, respectively, which is an improvement of 4.4% in terms of accuracy, 2.8% in terms of recall, and 2.7% in terms of mAP compared with the baseline algorithm. The improved YOLOv8 algorithm has a high detection accuracy, low leakage rate, and a detection speed of 107.5 FPS, which aligns with the real-time defect detection speed in the industry.