Many dyeing and finishing factories generally use old-fashioned dyeing machines. A key issue when using these machines is that the dyeing tank cannot detect entanglement problems, which may result in a lower dyeing quality. In this paper, imbalanced data with ensemble machine learning, such as Extreme Gradient Boosting (XGBoost) and random forest (RF), are integrated to predict the possible states of a dyeing machine, including normal operation, entanglement warning, and entanglement occurrence. To verify the results obtained using the proposed method, we worked with industry−academia collaborators. We collected 1,750,977 pieces of data from 1848 batches. The results obtained from the analysis show that after employing the Borderline synthetic minority oversampling technique and the Tomek link to deal with the data imbalance, combined with the model established by XGBoost, the prediction accuracy of the normal operation states, entanglement warning, and entanglement occurrence were 100%, 94%, and 96%, respectively. Finally, the proposed entanglement detection system was connected with the factory’s central control system using a web application programming interface and machine real-time operational parameter data. Thus, a real-time tangle anomaly warning and monitoring system was developed for the actual operating conditions.