In order to improve the accuracy of fabric texture defect detection, a combination of algorithm with Grabcut and convolutional neural network is proposed. Firstly, a segmentation algorithm based on grab-cuts is used to locate and segment the defects in fabric images accurately. Secondly, the sample images are expanded to increasing the number of the training samples. And then, the convolutional neural network is optimized to learn the features of the fabric defects more efficiently, which make it suitable for fabric defect recognition and classification. Experimental results shows that compared with other traditional algorithms, our model gets better performance with high accuracy of fabric defect detection.