Garment defect detection has been successfully implemented in the quality quick response system for textile manufacturing automation. Defects in the production of textiles waste a lot of resources and reduce the quality of the finished goods. It is challenging to detect garment defects automatically because of the complexity of images and variety of patterns in textiles. This study presented a novel deep learning-based Garment defect detection framework named as Deep-GD model for sequentially identifying image defects in patterned garments and classify the defect types. Initially, the images are gathered from the HKBU database and bilateral filters are used in the pre-processing of images to remove distortions. A squeeze-and-excitation network (SE-net) combined with Random Decision Forest Classifier with Bayesian Optimization (RDF-BO) algorithm is used to detect and classify garment defects. By analyzing the differences among the original and reconstruction images, the unsupervised technique trains to rebuild the fabric pattern. The SE-net is used to identify certain fabric flaws in the garments of the pre-processed images. Then, the defects-related garments are processed using RDF-BO algorithm for classifying the garment defect from these regions. Finally, the proposed Deep-GD model is used for classifying defected fabric into 12 classes such as Defect free, soil stain, oil stain, Double end, Snarls, Miss, Horizontal stripes, Lumpy, Dye spot, Fall out, Hairiness, tiny hole. The proposed Deep-GD model achieves the overall classification accuracy of 97.16%, which is comparatively superior to the existing techniques.