Aiming at the defect inspection under the characteristics of scale change, high reflection, inclined deformation of defects of lead bars and meeting the needs for faster detection, this paper proposes a faster and lighter cross-scale feature aggregation network (FLCNet). In this study, we focus on the redundancy of channel information, and design a new partial channel group convolution, based on which we design a Faster C3 module and a lightweight cross-scale feature fusion module. In addition, we design a cross-scale slim neck to reduce the redundant feature transfer of the model. Finally, we propose a uniform brightness acquisition method for lead bar sidewall image by using combined light source and construct a lead bar dataset with various complex defect samples. Experiments show that FLCNet effectively improves the detection accuracy of the surface defects of lead bars, the mAP@0.5 value reaches 97.1%, and compared with YOLOv5s, the model’s parameters reduced by 33.9%. At the same time, the detection speed reaches 114.9 FPS, which is faster than other advanced detection models.