Fabric quality plays a crucial role in modern textile industry processes. How to detect fabric defects quickly and effectively has become the main research goal of researchers. The You Only Look Once (YOLO) series of networks have maintained a dominant position in the field of target detection. However, detecting small-scale objects, such as tiny targets in fabric defects, is still a very challenging task for the YOLOv4 network. To address this challenge, this paper proposed an improved YOLOv4 target detection algorithm: using a combined data augmentation method to expand the dataset and improve the robustness of the algorithm, obtaining the anchors suitable for fabric defect detection by using the k-means algorithm to cluster the ground truth box of the dataset, adding a new prediction layer in yolo_head in order to have a better effect on tiny target detection, integrating a convolutional block attention module into the backbone feature extraction network, and innovatively replacing the CIOU loss function with the CEIOU loss function to achieve accurate classification and localization of defects. Experimental results show that compared with the original YOLOv4 algorithm, the detection accuracy of the improved YOLOv4 algorithm for tiny targets has been greatly increased, the AP value of tiny target detection has increased by 12%, and the overall mean average precision (mAP) has increased by 3%. The prediction results of the proposed algorithm can provide enterprises with more accurate defect positioning, reduce the defect rate of fabric products, and improve their economic effect.