In this paper, we present a robust and reliable framework based on L0 gradient minimization (LGM) and the fuzzy c-means (FCM) method to detect various fabric defects with diverse textures. In our framework, the L0 gradient minimization is applied to process the fabric images to eliminate the influence of background texture and preserve sharpened significant edges on fabric defects. Then, the processed fabric images are clustered by using the fuzzy c-means. Through continuous iterative calculation, the clustering centers of fabric defects and non-defects are updated to realize the defect regions segmentation. We evaluate the proposed method on various samples, which include plain fabric, twill fabric, star-patterned fabric, dot-patterned fabric, box-patterned fabric, striped fabric and statistical-texture fabric with different defect types and shapes. Experimental results demonstrate that the proposed method has a good detection performance compared with other state-of-the-art methods in terms of both subjective and objective tests. In addition, the proposed method is applicable to industrial machine vision detection with limited computational resources.