Fabric defect detection is a challenging task in the fabric industry because of the complex shapes and large variety of fabric defects. Many methods have been proposed to solve this problem, but their detection speed and accuracy were very low. As a classic deep learning method and end-to-end target detection algorithm, YOLOv4 has evolved rapidly and has been applied in many industries, showing good performance. This paper proposes an improved YOLOv4 algorithm with higher accuracy for fabric defect detection, in which a new SPP structure that uses SoftPool instead of MaxPool is adopted. The improved YOLOv4 algorithm with three SoftPools can process the feature map effectively, which has a significant advantage in reducing the negative side effects of the SPP structure and improving the detection accuracy. The improved SPP structure is used by the three outputs of Backbone, and in order to ensure that the output can be inputted into the subsequent PANet successfully, the network structure is improved that a series of convolution layers after the SPP structure is added for reducing the channel numbers of feature map to an appropriate value. In addition, contrast-limited adaptive histogram equalization is adopted in advance to improve the image quality, which results in strong anti-interference abilities and can slightly increase the mAP. Experimental results show that, compared with the original YOLOV4, the improved YOLOv4 increases the mAP effectively by 6%, while the FPS only decreases by 2. The improved YOLOv4 can identify the location of defects accurately and quickly, and can also be applied in other defect detection industries.