Evaluation of prediction and classification performances in different machine learning models for patient‐specific quality assurance of head‐and‐neck VMAT plans

Kusunoki, Terufumi; Hatanaka, Shin–ichi; Hariu, Masatsugu; Kusano, Yohsuke; Yoshida, Daisaku; Katoh, Hiroyuki; Shimbo, Munefumi; Takahashi, Tetsuya

doi:10.1002/mp.15393

Cited by 17 publications

(16 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sensitivity(Recall): the correct prediction ratio in the number of positive classes. See Equation (3). Specificity: the proportion of the number of negative classes that the wrong prediction was positive.…”

Section: Model Establishment and Evaluationmentioning

confidence: 99%

“…The conventional specific dosimetric verification based on the phantom which includes the dose recalculation, data transmission, set up, beam delivery, and γ analysis. It is not only increases the workload of the medical physicist, but also delays the first treatment of the patients [3]. To improve the efficiency and safety of IMRT implementation, the treatment plan complexity parameters were used to predict those plans that are not pass before treatment [4].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Prediction of radiomics-based machine learning for specific dosimetric verification of pelvic intensity modulated radiotherapy

Chen

Zhu

et al. 2023

Preprint

View full text Add to dashboard Cite

Background: Machine learning (ML) and deep learning(DL) technology has been used widely in the quality assurance. Due to the complexity of intensity modulated radiotherapy(IMRT)technology, the implementation of patient-specific quality assurance (PSQA) before the treatment has become an essential part in the IMRT. Therefore, this paper is aim to establish the different machine learning classification predict models of gamma pass rates for specific dosimetric verification of pelvic IMRT plan which based on the radiomic features and to explore the best prediction model. Methods: Retrospective analysis of the 3D dosimetric verification results based on measurements with gamma pass rate criteria of 3%/2 mm and 10% dose threshold of 196 pelvic intensity-modulated radiotherapy plans was carried. Prediction models were established by extracting radiomic features data. Four machine learning algorithms, random forest, support vector machine, adaptive boosting and gradient boosting decision trees, were used to calculate the AUC value, sensitivity and specificity respectively. The classification performance of the four prediction models were evaluated. Results: The sensitivity and specificity of the random forest, support vector machine, adaptive boosting, and gradient boosting decision trees models were 0.93,0.85,0.93,0.96, and 0.38,0.69,0.46, and 0.46, respectively. The AUC values for the random forest model and the adaptive boosting model were 0.81 and 0.82, respectively, and the AUC values for the support vector machine and gradient boosting decision tree models were 0.87. Conclusions: Machine learning methods based on radiomics can be used to establish a prediction model of gamma pass rate for specific dosimetric verification of pelvic intensity modulated radiotherapy. The classification performance of support vector machine model and gradient boosting decision trees model is better than that of random forest model and adaptive boosting model.The prediction model for a specific site is helpful to improve the performance of the model.

show abstract

Section: Model Establishment and Evaluationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Prediction of radiomics-based machine learning for specific dosimetric verification of pelvic intensity modulated radiotherapy

Chen

Zhu

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Artificial intelligence approaches have grown rapidly to improve conventional workflows of radiotherapy through automation 7–9 . In the case of the QA process, recently, several studies reported an auto PSQA process 10–40 . In general, they can be grouped into two categories: one for predicting the GPR 10–30 and the other for detecting errors 31–38 …”

Section: Introductionmentioning

confidence: 99%

“…In the case of the QA process, recently, several studies reported an auto PSQA process 10–40 . In general, they can be grouped into two categories: one for predicting the GPR 10–30 and the other for detecting errors 31–38 …”

Section: Introductionmentioning

confidence: 99%

Virtual pretreatment patient‐specific quality assurance of volumetric modulated arc therapy using deep learning

et al. 2023

View full text Add to dashboard Cite

BackgroundAutomatic patient‐specific quality assurance (PSQA) is recently explored using artificial intelligence approaches, and several studies reported the development of machine learning models for predicting the gamma pass rate (GPR) index only.PurposeTo develop a novel deep learning approach using a generative adversarial network (GAN) to predict the synthetic measured fluence.Methods and materialsA novel training method called “dual training,” which involves the training of the encoder and decoder separately, was proposed and evaluated for cycle GAN (cycle‐GAN) and conditional GAN (c‐GAN). A total of 164 VMAT treatment plans, including 344 arcs (training data: 262, validation data: 30, and testing data: 52) from various treatment sites, were selected for prediction model development. For each patient, portal‐dose‐image‐prediction fluence from TPS was used as input, and measured fluence from EPID was used as output/response for model training. Predicted GPR was derived by comparing the TPS fluence with the synthetic measured fluence generated by the DL models using gamma evaluation of criteria 2%/2 mm. The performance of dual training was compared against the traditional single‐training approach. In addition, we also developed a separate classification model specifically designed to detect automatically three types of errors (rotational, translational, and MU‐scale) in the synthetic EPID‐measured fluence.ResultsOverall, the dual training improved the prediction accuracy of both cycle‐GAN and c‐GAN. Predicted GPR results of single training were within 3% for 71.2% and 78.8% of test cases for cycle‐GAN and c‐GAN, respectively. Moreover, similar results for dual training were 82.7% and 88.5% for cycle‐GAN and c‐GAN, respectively. The error detection model showed high classification accuracy (>98%) for detecting errors related to rotational and translational errors. However, it struggled to differentiate the fluences with “MU scale error” from “error‐free” fluences.ConclusionWe developed a method to automatically generate the synthetic measured fluence and identify errors within them. The proposed dual training improved the PSQA prediction accuracy of both the GAN models, with c‐GAN demonstrating superior performance over the cycle‐GAN. Our results indicate that the c‐GAN with dual training approach combined with error detection model, can accurately generate the synthetic measured fluence for VMAT PSQA and identify the errors. This approach has the potential to pave the way for virtual patient‐specific QA of VMAT treatments.

show abstract

“…An extensive line of research focused on developing non-learned treatment plan complexity metrics such as modulation complexity score, mean aperture displacement, or small aperture score and investigating their correlation with PSQA failure 14,15,16,17,18,19,20 . A large number of papers further extended these approaches by developing classical machine learning and deep learning models to predict the PSQA failure based on a vast array of the plan complexity metrics as well as other heuristic features 21,22,23,24,25,26,27,28 We compared the performance of our model with the results from two leading gradient boosted decision tree models in their CatBoost and XGBoost implementations 36,37 widely used for tabular data as well as to a non-learned complexity metric, mean MLC gap.…”

Section: Introductionmentioning

confidence: 99%

Patient-specific Quality Assurance Failure Prediction with Deep Tabular Models

Levin

Aravkin

Kim

2022

Preprint

View full text Add to dashboard Cite

Background: Patient-specific quality assurance (PSQA) is part of the standard practice to ensure that a patient receives the dose from intensity-modulated radiotherapy (IMRT) beams as planned in the treatment planning system (TPS). PSQA failures can cause a delay in patient care and increase workload and stress of staff members. A large body of previous work for PSQA failure prediction focuses on non-learned plan complexity measures. Another prominent line of work uses machine learning methods, often in conjunction with feature engineering. Currently, there are no machine learning solutions which work directly with multi-leaf collimator (MLC) leaf positions, providing an opportunity to improve leaf sequencing algorithms using these techniques. Purpose: To improve patient safety and work efficiency, we develop a tabular transformer model based directly on the MLC leaf positions (without any feature engineering) to predict IMRT PSQA failure. This neural model provides an end-to-end differentiable map from MLC leaf positions to the probability of PSQA plan failure, which could be useful for regularizing gradient-based leaf sequencing optimization algorithms and generating a plan that is more likely to pass PSQA. Method: We retrospectively collected DICOM RT PLAN files of 968 patient plans treated with volumetric arc therapy. We construct a beam-level tabular dataset with 1873 beams as samples and MLC leaf positions as features. We train an attention-based neural network FT-Transformer to predict the ArcCheck-based PSQA gamma pass rates. In addition to the regression task, we evaluate the model in the binary classification context predicting the pass or fail of PSQA. The performance was compared to the results of the two leading tree ensemble methods (CatBoost and XGBoost) and a non-learned method based on mean MLC gap. Results: The FT-Transformer model achieves 1.44% Mean Absolute Error (MAE) in the regression task of the gamma pass rate prediction and performs on par with XGBoost (1.53 % MAE) and CatBoost (1.40 % MAE). In the binary classification task of PSQA failure prediction, FT-Transformer achieves 0.85 ROC AUC (with CatBoost and XGBoost achieving 0.87 ROC AUC and the mean-MLC-gap complexity metric achieving 0.72 ROC AUC). %While AUC summarizes the performance of binary classification models for all prediction thresholds, the performance at specific thresholds that maximize the true positive rate of PSQA failure identification while maintaining low false positive rate is of interest for clinical applications. Moreover, FT-Transformer, CatBoost, and XGBoost all achieve 80\% true positive rate while keeping the false positive rate under 20%. Conclusions: We demonstrate that reliable PSQA failure predictors can be successfully developed based solely on MLC leaf positions. Our FT-Transformer neural network can reduce the need for patient rescheduling due to PSQA failures by 80% while sending only 20% of plans that would not have failed the PSQA for replanning. FT-Transformer achieves comparable performance with the leading tree ensemble methods while having an additional benefit of providing an end-to-end differentiable map from MLC leaf positions to the probability of PSQA failure.

show abstract

Evaluation of prediction and classification performances in different machine learning models for patient‐specific quality assurance of head‐and‐neck VMAT plans

Cited by 17 publications

References 51 publications

Prediction of radiomics-based machine learning for specific dosimetric verification of pelvic intensity modulated radiotherapy

Prediction of radiomics-based machine learning for specific dosimetric verification of pelvic intensity modulated radiotherapy

Virtual pretreatment patient‐specific quality assurance of volumetric modulated arc therapy using deep learning

Patient-specific Quality Assurance Failure Prediction with Deep Tabular Models

Contact Info

Product

Resources

About