To address the detection challenges in defective images of insulators in transmission lines, including tiny object size, significant scale variations, a wide variety of defects, and complex background interference. In this study, an improved insulator defect detection algorithm is proposed, based on the YOLOv8s framework and combining feature enhancement and deformable convolution techniques. Firstly, to address the image feature distortion problem caused by aerial photography, a deformable convolutional feature extraction module (DCFEM) is introduced, which is designed to enhance the model's ability to adapt to the local geometric deformation, so as to effectively recover the distorted feature information in the image. Moreover, to enhance the detection ability of the model for small objects, a small object feature enhancement module (SOFEM) is designed, which adopts an efficient multi-scale attention mechanism, and aims to enhance the feature extraction ability of small objects, improve the sensitivity to small-size defects, and improve the detection accuracy. Eventually, to optimize the computational efficiency of the model, the Average Pooling-Sparse Convolution-Batch Normalization module (ASCBN) is proposed. This module combines average pooling, sparse convolution and batch normalization techniques to achieve a lightweight model while maintaining a high level of feature extraction capability. Experimental results on the CPLID dataset show that the improved model achieves a 4.3 percentage point improvement in the mAP metric compared to YOLOv8s, and the number of parameters in the model is reduced by 10%. The proposed scheme not only improves the accuracy and efficiency of defect detection, but also reduces the demand for computational resources, thus providing a more reliable and efficient solution for insulator defect detection in practical applications.