Fabric inspection is a crucial process in the textile industry's quality control. Due to the varying structures, textures, geometric features, and spatial distributions of fabric defects, manual fabric inspection is costly and inefficient. Existing fabric defect detection algorithms struggle to strike a balance among efficiency, accuracy, applicability, and deployment costs. In this model, an efficient lightweight fabric defect detection and classification algorithm based on deep convolutional neural networks is proposed. First, the algorithm performs cluster analysis on the fabric defect dataset to ensure that prior boxes better recall objects with fabric defect geometries and spatial characteristics. Next is fusing the convolutional block attention module attention mechanism and Swin Transformer module with the CSPNet structure. This fusion enhances the model's focus on local features and its ability to capture global contextual information without sacrificing the model's inference speed. Moreover, WIoU or Wise-IoU is used as the bounding box loss function of the model, which improves the convergence speed of the bounding box loss and enhances the positioning ability of the model. Finally, the performance of the improved model was validated on a public dataset, showing varying degrees of improvement compared to the baseline model and other state-of-the-art algorithms, meeting the requirements of modern textile processes.