Defect segmentation has been a focal point in fabric inspection research, and remains challenging because it detects delicate features of defects complicated by variations in weave textures and changes in environmental factors (e.g., illumination, noise). Based on characteristics of fabric structure, an approach of using local contrast deviation (short for LCD) is proposed for fabric defect detection in this paper. LCD is a parameter to describe features of the contras difference in four directions between the analyzed image and a defect-free image of the same fabric, and used a threshold function for defect segmentation. The validation tests on the developed algorithms were performed with images from TILDA's Textile Texture Database, and comparing with an approach of using modified local binary patterns (short for LBP), experimental results show that the proposed method has robustness and simplicity.