The most challenging task in the cotton business is finding Fabric Faults (FFs) and refining material durability appropriately. To alleviate this, an Enhanced Pairwise-Potential Activation Layer in Optimized Multi-Criteria Convolutional Neural Network (EPPAL-OMCCNN) model was created, which considers a multi-objective active sampling strategy for annotation and tuning CNN for FF detection. But, it needs to predict historical and new kinds of unknown FF patterns accurately. So, this article introduces a deep Reinforcement Learning (RL) scheme into the EPPAL-OMCCNN model to predict new unknown FFs with the help of prior knowledge. At first, the multi-objective sampling strategy is applied to the fabric image database to label more influential images. Then, these images are used to construct the Optimized CNN (OCNN) with the RL model, which is trained by the fabric defect characteristics to predict the new unknown fabric pattern defects precisely. Finally, the experimental results exhibit that the EPPAL-OMCCNN-RL model on the TILDA set accomplishes 97.58% accuracy contrasted with the different deep learning-based FF detection models.