Insulator defect detection is an important task in inspecting overhead transmission lines. However, the surrounding environment is complex, and the detection accuracy of traditional image processing algorithms is low. Therefore, insulator defect detection is still mainly performed manually. In order to improve this situation, we proposed an insulator defect detection method called INSU-YOLO based on deep neural networks. Overexposure points in the image will interfere with insulator detection, so we used image augment to reduce noise and extract the edge information of the insulator. Based on an attention mechanism, we introduced a structure called attention-block where the backbone extracts the feature map, and this aims to improve the ability of our method to detect insulators. Insulators have a variety of specifications, and the location and granularity of defects are also different. Therefore, we proposed an adaptive threat estimation method based on the area ratio between the entire insulator and the defect area. In addition, in order to solve the problem of data shortage, we established a dataset called InsuDetSet for model training. Experiments on the InsuDetSet dataset demonstrated that our model outperforms existing state-of-the-art models regarding both the detection box and speed.