PurposeTreatment plans are used for patients under radiotherapy in clinics. Before execution, these plans are checked for safety and quality by human experts. A few of them were identified with flaws and needed further improvement. To automate this checking process, an unsupervised learning method based on an autoencoder was proposed.MethodsFirst, features were extracted from the treatment plan by human experts. Then, these features were assembled and used for model learning. After network optimization, a reconstruction error between the predicted and target signals was obtained. Finally, the questionable plans were identified based on the value of the reconstruction error. A large value of the reconstruction error indicates a longer distance from the standard distribution of normal plans. A total of 576 treatment plans for breast cancer patients were used for the test. Among them, 19 were questionable plans identified by human experts. To evaluate the performance of the autoencoder, it was compared with four baseline detection algorithms, namely, local outlier factor (LOF), hierarchical density-based spatial clustering of applications with noise (HDBSCAN), one-class support vector machine (OC-SVM), and principal component analysis (PCA).ResultsThe results showed that the autoencoder achieved the best performance than the other four baseline algorithms. The AUC value of the autoencoder was 0.9985, while the second one was 0.9535 (LOF). While maintaining 100% recall, the average accuracy and precision of the results by the autoencoder were 0.9658 and 0.5143, respectively. While maintaining 100% recall, the average accuracy and precision of the results by LOF were 0.8090 and 0.1472, respectively.ConclusionThe autoencoder can effectively identify questionable plans from a large group of normal plans. There is no need to label the data and prepare the training data for model learning. The autoencoder provides an effective way to carry out an automatic plan checking in radiotherapy.