<p>With the advent of the Industrial 4.0 era, deep learning has been continuously applied to the task of surface defect detection, and effective progress has been made. However, the limited number of training samples and high labelling costs are considerable obstacles to the vigorous development of this task. Thus, we explore the use of different numbers of labels with various accuracies during training to achieve the maximum detection accuracy with the lowest cost. Our proposed method includes improved segmentation and decision networks. An attention mechanism is integrated into the segmentation subnetwork. Moreover, atrous convolutions are used in the segmentation and decision subnetworks. In addition, the original loss function is improved. Several experiments are carried out on the Severstal Steel Defect dataset collected in Germany, and the results show that each component improves the detection accuracy by 1% to 2%. Finally, when we add an appropriate number of pixel-level labels in the weakly supervised learning mode, the detection accuracy reaches that of the fully supervised mode with a significantly reduced annotation cost.</p> <p>&nbsp;</p>