For replacing error-prone, costly, incongruous, and time-consuming human operators in the textile industry, an automatic and precise fabric flaw inspection system is vital. Textile blemish detection plays an essential effect in the production of textiles in the textile industry. Researchers have proposed many algorithms for fabric defect detection. However, several important issues, including inference time consumption, data imbalance, and algorithm complexity, still need to be dealt with for application in textile industry production. This paper proposes a two-model automatic towel defect detection method based on YOLOv5 to detect the defects of towels, effectively improving the detection efficiency and significantly reducing the cost. Moreover, this method has achieved the detection speed of 10 pictures per second in the actual production.