Surface inspection is a necessary process of fabric quality control. However, it remains a challenging task owing to diverse types of defects, various patterns of fabric texture, and application requirements for detection speed. In this paper, a lightweight deep learning model is therefore proposed to complete the segmentation of fabric defects. The input of the model is a fabric image, and the output is a binary image. Generally known, a deep learning model usually needs much data to update the parameters. Still, as an abnormal phenomenon, fabric defects are unpredictable, which makes it impossible to collect a large number of data. Distinct from other models, the proposed method is a supervised network but does not need manually labeled samples for training. A fake sample generator is designed to simulate the defect image, which only needs the defect-free fabric image. The proposed model is trained with fake samples and verified with real samples. The experimental results show that the model trained with false data is useful and achieves high segmentation accuracy on real fabric samples. Besides, a loss function is proposed to deal with the problem of imbalance between the number of background pixels and the number of defective pixels in the fabric image. Comprehensive experiments were performed on representative fabric samples to verify the segmentation accuracy and detection speed of this method. INDEX TERMS Fabric defect; deep learning; image segmentation; defect detection; imbalanced dataset.