A risk assessment of automated treatment planning and recommendations for clinical deployment

Abstract

Purpose: To assess the risk of failure of a recently developed automated treatment planning tool, the Radiation Planning Assistant (RPA) and to determine the reduction in these risks with implementation of a quality assurance (QA) program specifically designed for the RPA. Methods: We used failure mode and effects analysis (FMEA) to assess the risk of the RPA. The steps involved in the workflow of planning a 4-field box treatment of cervical cancer with the RPA were identified. Then, the potential failure modes at each step and their causes were identified and scored according to their likelihood of occurrence, severity, and likelihood of going undetected. Additionally, the impact of the components of the QA program on the detectability of the failure modes was assessed. The QA program was designed to supplement a clinic’s standard QA processes and consisted of 3 components: (1) automatic, independent verification of the results of automated planning; (2) automatic comparison of treatment parameters to expected values; and (3) guided manual checks of the treatment plan. A risk priority number (RPN) was calculated for each potential failure mode with and without use of the QA program. Results: In the RPA automated treatment planning workflow, we identified 68 potential failure modes with 113 causes. The average RPN was 91 without the QA program and 68 with the QA program (maximum RPNs were 504 and 315, respectively). The reduction in RPN was due to an improvement in the likelihood of detecting failures, resulting in lower detectability scores. The top-ranked failure modes included incorrect identification of the marked isocenter, inappropriate beam aperture definition, incorrect entry of the prescription into the RPA plan directive, and lack of a comprehensive plan review by the physician. Conclusions: Using FMEA, we assessed the risks in the clinical deployment of an automated treatment planning workflow and showed that a specialized QA program for the RPA, which included automatic QA techniques, improved the detectability of failures, reducing this risk. However, some residual risks persisted, which were similar to those found in manual treatment planning, and human error remained a major cause of potential failures. Through the risk analysis process, we identified 3 key aspects of safe deployment of automated planning: (1) user training on potential failure modes; (2) comprehensive manual plan review by physicians and physicists; and (3) automated QA of the treatment plan.

Generating High-Quality Lymph Node Clinical Target Volumes for Head and Neck Cancer Radiation Therapy Using a Fully Automated Deep Learning-Based Approach

“…For example, the Radiation Planning Assistant (RPA, https:// rpa.mdanderson.org) is being developed at The University of Texas MD Anderson Cancer Center to fully automate the radiation treatment planning process with no or minimal user interventions. [45][46][47][48] For such a system to be effective in reducing the workload at busy clinics with limited resources, including radiation oncologists, high-quality automatic contouring is essential. 23 The present study showed that deep learning can achieve this, and we are integrating these tools to autosegment normal tissue and target volumes into the RAP system.…”

Automatic Verification of Beam Apertures for Cervical Cancer Radiation Therapy

“…In a previous risk assessment of automated treatment planning using failure modes and effects analysis, it was found that beam aperture creation was 1 of the high-risk areas subject to failure. 25 Currently, standard practice relies solely on 1 physician to determine the clinical acceptability of the beam apertures; there is no secondary check by an independent expert. To our knowledge, the QA technique presented in this work is the first technique to automatically verify the clinical acceptability of beam apertures.…”

The Emergence of Artificial Intelligence within Radiation Oncology Treatment Planning

“…This dual expert approach performs treatment planning tasks (e.g., contouring, isocenter detection, field aperture creation) using independent models, and flags erroneous contours or plans based on metrics which quantify agreement between the two methods. Such an approach has been found to be effective in increasing the detectability of treatment planning failures when assessed by FMEA for cervix and ROC analysis for the head and neck [45, 46].…”