Surface defect detection is an important task in industrial production. Although significant progress has been made in deep learning-based defect segmentation methods, the low contrast between defect and background and the shape and scale diversity of defects limit the models' detection accuracy and generalization ability. Therefore, realizing the full automation of surface defect detection still faces many challenges. To overcome these problems, this paper proposes a skeleton-strengthening network called SKS-Net, which provides stable and excellent surface defect detection performance even at low contrast and multi-scale. SKS-Net designs a skeleton-strengthening convolutional module to capture multi-scale features efficiently. The convolution kernel of this convolution module is closer to the shape of the segmentation target, which significantly reduces irrelevant regions and improves the feature extraction capability of the convolution kernel. This convolution module can be directly embedded into existing network structures without adding additional computational overhead. In addition, we design a new feature channel fusion module to extract key information from features at different levels. To improve the training effect, we introduce a multi-scale auxiliary supervision mechanism. The proposed model is evaluated on four different publicly available surface defect datasets and compared with other state-of-the-art models. Results show that SKS-Net performs exceptionally well in terms of accuracy, achieving 66.72% mIoU on the KolektorSDD dataset. The code is publicly available at https://github.com/Wanglaoban3/SKS-Net.git.