The development of industry is inseparable from the support of steel materials, and the modern industry has increasingly high requirements for the quality of steel plates. But the process of steel plate production produces many types of defects, such as roll marks, scratches, and scars. These defects will directly affect the quality and performance of the steel plate, so it is necessary to effectively detect them. Steel plate surface defects are characterized by their types, shape, and size: the same defect can have different morphologies, and similarities can exist between different defects. In this paper, industrial steel plate surface defect samples are analyzed, and a sample set is established by screening the collected defect images. Then, annotation and classification are performed. A multilayer feature extraction framework is developed in experiments to train a neural network on the sample set of defects. To address the problems of low automation, slow detection speed, and low accuracy of the traditional defect detection methods, the attention graph convolution network (AGCN) is investigated in this paper. Firstly, faster R-CNN is used as the basic network model for defect detection, and the visual features are jointly refined by combining attention mechanism and graph convolution neural network. The latter network enriches the contextual information in the visual features of steel plates and explores the semantic association between vision and defect types for different kinds of defects using the attention mechanism to achieve intelligent detection of defects, thus enabling our method to meet the practical needs of steel plate production.