Due to the complex diversity of both fabric texture and defect, fabric defect detection is a challenge topic. However, most of existing defect detection methods can still detect only one type of fabric defects. In order to solve this problem, we propose a universal and adaptive defect detection algorithm based on dictionary learning for detecting various defects of different fabric texture. Firstly, in order to make the image more balanced and improve the detection accuracy, according to the complexity of fabric texture and the degree of brightness and darkness of the background, we segment defect-free image according to a certain size to obtain the image joint matrix. In order to make the algorithm more universal, next we form a random dictionary instead of over-complete dictionary and fixed dictionary by randomly select feature columns from the image joint matrix, which can extract the most important texture and background information of fabric. Then, we use the Orthogonal Matching Pursuit algorithm to obtain the sparse representation of defect-free image. On this basis, we use the K-Singular Value Decomposition algorithm to update the dictionary iteratively to get the final dictionary. Afterward, the defective image is reconstructed with the learned dictionary to obtain the reconstruction error. Finally, the correction coefficient is obtained through reconstruction error analysis, then the corresponding adaptive threshold is used to detect and mark the location of the defects. The performance of the algorithm is evaluated on the fabric database of raw fabric, yarn-dyed fabric and patterned fabric. The experimental results show that the proposed algorithm has good universality, adaptability and a high detection success rate for different types of fabrics. Particularly, the average detection success rate of dark-red fabric and dot-patterned fabric can reach 100%, and the Precision and the Recall of defect detection evaluation index can also reach 100%.