The importance of evaluating the complete automated knowledge-based planning pipeline

Abstract

We determine how prediction methods combine with optimization methods in two-stage knowledge-based planning (KBP) pipelines to produce radiation therapy treatment plans. We trained two dose prediction methods, a generative adversarial network (GAN) and a random forest (RF) with the same 130 treatment plans. The models were applied to 87 out-of-sample patients to create two sets of predicted dose distributions that were used as input to two optimization models. The first optimization model, inverse planning (IP), estimates weights for dose-objectives from a predicted dose distribution and generates new plans using conventional inverse planning. The second optimization model, dose mimicking (DM), minimizes the sum of one-sided quadratic penalties between the predictions and the generated plans using several dose-objectives. Altogether, four KBP pipelines (GAN-IP, GAN-DM, RF-IP, and RF-DM) were constructed and benchmarked against the corresponding clinical plans using clinical criteria; the error of both prediction methods was also evaluated. The best performing plans were GAN-IP plans, which satisfied the same criteria as their corresponding clinical plans (78%) more often than any other KBP pipeline. However, GAN did not necessarily provide the best prediction for the second-stage optimization models. Specifically, both the RF-IP and RF-DM plans satisfied all clinical criteria 25% and 15% more often than GAN-DM plans (the worst performing planning), respectively. GAN predictions also had a higher mean absolute error (3.9 Gy) than those from RF (3.6 Gy). We find that state-of-the-art prediction methods when paired with different optimization algorithms, produce treatment plans with considerable variation in quality.

Artificial intelligence and machine learning for medical imaging: A technology review

“…Nevertheless, the work of Deist et al included one of the largest datasets investigated so far for radiotherapy outcome prediction, which is a strong argument in favor of considering RFs as one of the first options to investigate for this kind of application. In addition, RFs keep achieving very promising results in recent applications related to outcome prediction [135,[139][140][141][142][143], but also for other domains like image classification [113,144] or automatic treatment planning [100,[145][146][147]. Regarding other tasks where RFs were among the state-of-the-art methods a few years ago, like image synthesis [148][149][150] or segmentation [151,152], the community has now fully switched the attention to CNNs [5,153,154].…”

Probabilistic feature extraction, dose statistic prediction and dose mimicking for automated radiation therapy treatment planning

“…Compared to the methods in early studies on knowledge-based planning [22][23][24], instead of predicting principal component coefficients, our method outputs complete probability distributions capable of modeling inter-statistic dependencies, skewness and eventual multimodality. As argued by Babier et al [28], it is crucial to maintain a holistic perspective when developing and assessing constituent methods in a prediction-mimicking pipeline-there is, for instance, no direct causality between the performance of a dose prediction model and the quality of the produced plan. In this sense, the proposed division of the pipeline into feature extraction, dose statistic prediction and dose mimicking has the advantages of being flexible, with each part being substitutable with any other algorithm, and general, with minimal loss of information between the steps.…”

OpenKBP: The open‐access knowledge‐based planning grand challenge and dataset

“…In most cases, the first stage is a machine learning (ML) method that predicts the dose distribution that should be delivered to a patient based on contoured CT images, and the second stage is an optimization model that generates a treatment plan based on the predicted dose distribution. 3,4 Research into dose prediction has experienced major growth in the past decade, 5 in part due to the growing sophistication of machine learning and optimization methods in conjunction with advances in computational technology. There are two main branches of dose prediction methods: (a) those that predict summary statistics (e.g., dose-volume features) [6][7][8][9] and (b) those that predict entire three-dimensional (3D) dose distributions.…”

“…1). In most cases, the first stage is a machine learning (ML) method that predicts the dose distribution that should be delivered to a patient based on contoured CT images, and the second stage is an optimization model that generates a treatment plan based on the predicted dose distribution 3,4 …”

“…Researchers that develop dose prediction models must rely on their own private clinical datasets and different evaluation metrics, which makes it difficult to objectively and rigorously compare the quality of different prediction approaches at a meaningful scale 5 . As a result, researchers must attempt to reproduce published dose prediction models to benchmark their new models via a common dataset and set of evaluation metrics 4,19 . In contrast, open‐access datasets and standardized metrics are staples of thriving artificial intelligence‐driven fields, as evidenced by the uptake of the CIFAR 20 and ImageNet 21 datasets in the computer vision community over the past decade.…”