In view of the complex multi-scale target detection environment of ultrasonic atlas of weld defect and the poor detection performance of existing algorithms for the multiple small target defects, the Faster RCNN convolution neural network is applied to weld defect detection, and a Fast RCNN deep learning network is proposed in combination with an improved ResNet 50. Based on the coexistence of multiple small targets and multi-scale target detection, this paper proposes to combine deformable network, FPN network and ResNet50 to improve the detection performance of the algorithm for multi-scale targets, especially small targets. Based on the efficiency and accuracy of candidate frame selection, K-means clustering algorithm and ROI Align algorithm are proposed, and the anchors points and candidate frames suitable for weld defect data sets are customized for accurate positioning. Through the self-made ultrasonic atlas data set of weld defects and experimental verification of the improved algorithm in this paper, the overall mean average precision has reaches 93.72%, and the average precision of small target defects such as “stoma” and “crack” has reaches 92.5% and 88.9% respectively, which is 4.8% higher than the original Faster RCNN algorithm. At the same time, through the ablation experiments and comparison experiments with other mainstream target detection algorithms, it is proved that the improved method proposed in this paper improves the detection performance and is superior to other algorithms. The actual industrial detection scene proves that it basically meets the requirements of weld defect detection, and can provide a reference for the intelligent detection method of weld defects.