Focusing on the fabric defect detection with periodic-pattern and pure-color texture, an algorithm based on Direction Template and Image Pyramid is proposed. The detection process is divided into two stages: model training and defect localization. During the model training stage, we construct an Image Pyramid for each fabric image that does not contain any defects. Then, Stacked De-noising Convolutional Auto-Encoder (SDCAE) is used for image reconstruction, its training sets are created by randomly extracting image blocks from image pyramid, which makes the feature information of the image block more abundant and the reconstruction effect of the model more remarkable. During the defect localization stage, the image to be detected is divided into a number of blocks, and is reconstructed by using the trained SDCAE model. Then, the candidate defective image blocks are roughly located by using the Structural Similarity Index Measurement after the image reconstruction. Subsequently, direction template is introduced to solve the problem of fabric deformation caused by factors such as fabric production environment and photographic angle. We select the direction template of the images to be detected, filter the candidate defective blocks, and further reduce false detection rate of the proposed algorithm. Furthermore, there is no need to calculate size of periodic-pattern during detection for periodic textured fabric. The algorithm is also suitable for defect detection for pure-color fabrics. The experimental results show that the proposed algorithm can achieve better defect localization accuracy, and receive better results in detection of pure-color fabrics, compared with traditional methods. INDEX TERMS Fabric defect detection, direction template, image pyramid, stack de-noising convolutional auto-encoder, similarity measure.