The essence of the difficulties for weld surface detection is that there is a lot of interference information during detection. This study aims to enhance the detection accuracy while keeping great deployment capabilities of a detection model for weld surface defects. To achieve this goal, an improved Yolo-GCH model is proposed based on the stable and fast Yolo-v5. The improvements primarily involve introducing a graph convolution network combined with a self-attention mechanism in the head part (i.e., GCH). This component focuses on improving the insufficient recognition capability of CNN for similar defects in complex environments. Furthermore, to address the presence of potentially ambiguous samples in complex welding environments, the label assignment strategy of simOTA is implemented to optimize the anchor frame. Additionally, a streamlined structure, aiming to improve model detection speed while minimizing performance impact, has been designed to enhance the applicability of the model. The results demonstrate that the cooperation of GCH and simOTA significantly improves the detection performance while maintaining the inference speed. These strategies lead to a 2.5% increase in mAP@0.5 and reduce the missing detection rates of weld and 8 types of defects by 32.9% and 84.1% respectively, surpassing other weld surface detection models. Furthermore, the impressive applicability of the model is verified across four scaled versions of Yolo-v5. Based on the proposed strategies, the FPS increases by more than 30 frames in the fast s and n versions of Yolo-v5. These results demonstrate the great potential of the model for industrial applications.