The precise location of insulators in infrared images is of great significance for insulator condition monitoring and fault diagnosis. Due to the characteristics of insulators themselves and the use of handheld infrared cameras, insulators usually appear in infrared images with different aspect ratios and main axis orientations. Therefore, it is very important and necessary to make full use of the prior knowledge of the insulator itself to accurately locate it. However, most of the existing methods use axial horizontal detection boxes to detect insulators, which cannot take into account the characteristics of the insulator well. When there are large overlapping areas of two horizontal detection boxes, the non-maximum suppression algorithm may lead to missed detection of the object. In order to further improve the accuracy of the detection algorithm, this paper makes full use of the prior features carried by the insulator itself, and optimizes Faster R-CNN from five aspects: rectangular box representation, feature extraction, candidate box generation, anchor design, and feature alignment. An oriented detection model for infrared images of insulators is constructed. Comparative experiments with a variety of mainstream detection methods were carried out on the constructed infrared dataset. The results show that the proposed method is superior to other models in detection accuracy. When the intersection and union ratio is 0.5, the average precision reaches 95.08%. In addition, it can also effectively predict the shape and angle information of insulators in complex scenes, laying a beneficial foundation for subsequent diagnosis automation tasks.