Fabric defect detection is a crucial aspect of the textile industry. Currently, deep learning methods have demonstrated exceptional performance in fabric defect detection tasks. However, their performance is greatly affected by the number of defect samples, which is a challenge to obtain during in actual production. To address this issue, this paper proposes an unsupervised anomaly detection method for fabric defects using image reconstruction networks. This method only requires defect‐free samples for training. During the training phase, the model compresses defect‐free samples to obtain a low‐dimensional manifold and reconstruct them. During the inference phase, the method assesses whether a sample is defective by calculating the reconstruction error between the input and output images, and locates the defect region by computing the difference in various patches. Furthermore, since fabric contains rich texture features, with high correlation between neighboring pixels, a structure similarity index measure combined with mean absolute error is introduced to evaluate the reconstruction error, which enhances the model's representation ability for defect‐free samples. Additionally, considering the diverse texture backgrounds in fabric, a multiscale reconstruction module is designed to optimize the reconstruction effect. Experimental results demonstrate that compared with other related approaches, the proposed method achieves high accuracy (Image‐based AUC up to 98.2% and pixel‐based AUC up to 97.3%) on multiple datasets and has good generalization ability for different fabric textures.This article is protected by copyright. All rights reserved.