Insulators have an extremely important role in transmission lines, and they are important components for ensuring the safe operation of transmission lines. In order to solve the difficult problem of insulator fault detection under complex background, IG‐YOLOv8 insulator fault detection algorithm is proposed in this paper. First, the Wise‐IoU (WIoU) loss function is introduced to mitigate the adverse impact of low‐quality images by employing a dynamic non‐monotonic focusing mechanism, thereby enhancing the detection performance of the entire model. Second, a novel C2f network is constructed by integrating the receptive field coordination attention (RFCA) convolutional module to address the parameter‐sharing issue associated with large convolutional kernels. Additionally, the data set has been reorganized using k‐fold cross‐validation to ensure that each subset undergoes training and testing, consequently reducing generalization errors. Finally, a deformable attention (DA) mechanism is employed to augment the feature extraction capability pertaining to insulator fault region information. In order to evaluate the detection performance of the improved IG‐YOLOv8 algorithm, this study constructed an insulator target detection data set containing four fault types: Normal, Defect, Dirty, and Aging. The experimental results show that the average accuracy of the improved model is increased from 89.7% to 96.9%, and the Recall value of the Aging type insulator is increased from 71.8% to 89.1%. The occurrence of missed detection is greatly reduced, and the accuracy of detection is improved. © 2024 Institute of Electrical Engineers of Japan and Wiley Periodicals LLC.