A STAIN defect is difficult to detect with the naked eye because of its characteristic of having a very minimal difference in brightness with the local area of the surface. Usually, background extraction–based and Gabor filter–based texture feature–based methods have been proposed to detect STAIN defects. Recently, with the development of deep learning, the convolutional neural network (CNN)–based detection method has been proposed. Gabor filter images have an advantage in image texture analysis and can be used as an input to the CNNs. However, because the original image and the Gabor filter image have different domains, it is necessary to design a CNN that can extract features advantageous for STAIN defect classification. In this paper, a CNN for STAIN defect classification using the original image and Gabor filter image was proposed. A Gary-Level Co-occurrence Matrix (GLCM) matrix from the Gabor filter image was constructed, and then the Inverse Difference Moment (IDM) value, which is a homogeneity local feature, was extracted. Gabor filter images suitable for STAIN defect classification were selected using IDM values, and a STAIN defect detection system that uses the selected Gabor filter images and the inspection image as an input to the CNN was constructed. As a result of the experiment based on the magnetic tile data set and the compact camera module (CCM) dataset, it was confirmed that the proposed method has an improved performance based on the precision, recall, and F1-score in comparison to the single-stream CNN-based method.