Convolution neural network (CNN) is one of the most popular machine learning techniques that is being used in many applications like image classification, image analysis, textile archives, object recognition, and many more. In the textile industry, the classification of defective and nondefective fabric is an essential and necessary step to control the quality of fabric. Traditionally, a user physically inspects and classifies the fabric, which is an ineffective and tedious activity. Therefore, it is desirable to have an automated system for detecting defects in the fabric. To address these issues, this research proposes a solution for classifying defective and nondefective fabric using deep learning-based framework. Therefore, in this research, an image processing technique with CNN-based GoogleNet is presented to classify defective and nondefective fabric. To achieve the purpose, the system is trained using different kinds of fabric defects. The performance of the proposed approach was evaluated on the textile texture TILDA dataset, and achieved a classification accuracy of 94.46%. The classification results show that the proposed approach for classifying defective and nondefective fabric is better as compared to other state-of-the-art approaches such as Bayesian, BPNN, and SVM.