Characterization of a Bayesian network‐based radiotherapy plan verification model

Abstract

Purpose The current process for radiotherapy treatment plan quality assurance relies on human inspection of treatment plans, which is time‐consuming, error prone and oft reliant on inconsistently applied professional judgments. A previous proof‐of‐principle paper describes the use of a Bayesian network (BN) to aid in this process. This work studied how such a BN could be expanded and trained to better represent clinical practice. Methods We obtained 51 540 unique radiotherapy cases including diagnostic, prescription, plan/beam, and therapy setup factors from a de‐identified Elekta oncology information system from the years 2010–2017 from a single institution. Using a knowledge base derived from clinical experience, factors were coordinated into a 29‐node, 40‐edge BN representing dependencies among the variables. Conditional probabilities were machine learned using expectation maximization module using all data except a subset of 500 patient cases withheld for testing. Different classes of errors that were obtained from incident learning systems were introduced to the testing set of cases which were withheld from the dataset used for building the BN. Different sizes of datasets were used to train the network. In addition, the BN was trained using data from different length epochs as well as different eras. Its performance under these different conditions was evaluated by means of Areas Under the receiver operating characteristic Curve (AUC). Results Our performance analysis found AUCs of 0.82, 0.85, 0.89, and 0.88 in networks trained with 2‐yr, 3‐yr 4‐yr and 5‐yr windows. With a 4‐yr sliding window, we found AUC reduction of 3% per year when moving the window back in time in 1‐yr steps. Compared to the 4‐yr window moved back by 4 yrs (2010–2013 vs 2014–2017), the largest component of overall reduction in AUC over time was from the loss of detection performance in plan/beam error types. Conclusions The expanded BN method demonstrates the ability to detect classes of errors commonly encountered in radiotherapy planning. The results suggest that a 4‐yr training dataset optimizes the performance of the network in this institutional dataset, and that yearly updates are sufficient to capture the evolution of clinical practice and maintain fidelity.

Toward automation of initial chart check for photon/electron EBRT: the clinical implementation of new AAPM task group reports and automation techniques

“…In assessing the status of automation tool development, it seems likely that lower dimensional problems, such as treatment parameter comparison, can be easily handled by scripts/programs. Higher dimensional problems in physician order error, including disease staging and treatment modality decision, may be taken care of by machine learning, such as a k‐means clustering algorithm, 8 random forest methods, 37 or Bayesian networks as proposed by Kalet et al 38 and further developed by Luk et al 31 As Kalet et al 39 and Pallai et al 40 pointed out, machine learning still faces many challenges and must be quality assured before introduction into the clinic. The breakthrough of automation tools or machine learning beyond low‐level checks will take some time.…”

“…Auto_UMMS_Exp can help to achieve 6%, 18%, and 46% in corresponding reductions. forest methods, 37 or Bayesian networks as proposed by Kalet et al 38 and further developed by Luk et al 31 As Kalet et al 39 and Pallai et al 40 pointed out, machine learning still faces many challenges and must be quality assured before introduction into the clinic. The breakthrough of automation tools or machine learning beyond low-level checks will take some time.…”

“…[22][23][24][25][26][27][28][29][30] Some items previously deemed impossible for automation may become feasible through the development of new and easy-to-implement machine learning techniques. One example is a study by Luk et al 31 who, using a Bayesian network-based radiotherapy plan verification model, suggested a 4-year training dataset to optimize the performance of the network, and yearly updates were considered sufficient to capture the evolution of clinical practice and maintain fidelity. Other recently published papers describe and quantify time savings and error reduction using different analyses.…”

Graph‐based risk assessment and error detection in radiation therapy

“…Such a model also promotes standardization. Our solution differs from existing automated solutions in the literature for the TPCR that do not provide contextual information about errors or recent Bayesian network‐based methods where probabilistic relationships among a subset of variables are learnt from prior treatment plans 20–22 …”

Ontologies in radiation oncology

“…In this section we present such an example using a dependency layered ontology for radiation oncology (DLORO). The ontology was initially designed to automate the construction of error detection Bayesian networks (BN) [38][39][40][41] for radiotherapy treatment plans but as an ontology, it also contains the semantic properties needed to map radiation oncology concepts and terms in the models to the schemas of relational databases. Here we map the DLORO to the two major oncology information systems (OIS), Aria (Varian Medical System, Palo Alto, USA) and Mosaiq (Elekta AB, Stockholm, Sweden).…”