Reinforcement Learning Strategies for Decision Making in Knowledge-Based Adaptive Radiation Therapy: Application in Liver Cancer

Abstract

Purpose/Objective(s): Treatment choices in adaptive radiotherapy are often based on subjective experiences and heuristic rules that lack an effective strategy to dynamically optimize long-term clinical outcomes. Therefore, we are investigating reinforcement learning (RL) approaches that can provide a robust framework to interactively adapt treatment regimens to individual patient's characteristics over time. In this context, RL approaches would balance trade-offs between exploring varying dose fractionation options and exploiting knowledge synthesized from patient's time-dependent covariates (states) history to maximize accumulative reward (TCP) and minimize long-term regret (NTCP). Materials/Methods: We studied an institutional cohort of 145 hepatocellular carcinoma SBRT patients with 45 on non-adaptive and 100 on adaptive protocols. Adaptation was based on liver function, assessed with Indocyanine green retention, by delivering a split-course of 3 + 2 fractions with a month break in between. Median prescribed tumor dose was 40 Gy delivered in 3 or 5 fractions with doses converted into EQD2 using the LQL model. Plasma biomarkers were analyzed before and during treatment. NTCP was assessed as a 1-grade change in ALBI score. The RL radiotherapy environment was modeled as a 2-stage Markov decision process of baseline and one month into treatment states. States are represented by the patient's clinical, dosimetric, and biological covariates history. Two typical decision-making scenarios at stage-2 were considered for testing RL feasibility retrospectively: (1) choosing to adapt with a splitcourse or not, and (2) determining to deliver an additional 2 fractions after the initial course. The reward/regret was defined by the complication-free tumor control (P+ Z TCP Â (1-NTCP)). Q-learning with a simple regression model of state-action mapping was used for optimizing strategy selection by solving the corresponding Bellman equation. The performance was evaluated using an adjusted R-squared (aR 2 ) to correct for over-fitting pitfalls. Results: Using states of clinical and dosimetric (tumor size, tumor dose, mean liver dose) covariates, Q-learning at one month (stage-2) selected split-course adaptation as an optimal action with a model fit of aR 2 Z 0.65 (P < 0.001). Percentage change in TGF-b1 concentration was the only biological variable to correlate with outcomes (ALBI score, P Z 0.03); its addition improved the fit to aR 2 Z 0.74. In case of 3 versus 5 fractions determination, the delivery of 2 extra fractions was found to be a better action with an aR 2 Z 0.66 (P < 0.001). The inclusion of TGF-b1 improved the fit to aR 2 Z 0.74. Conclusion:Our results demonstrate that RL approaches provide a promising framework for sequential clinical decision making in adaptive radiotherapy. Moreover, biological metrics seem to improve the goodness of fit. We are exploring advanced Q-learning with nonlinear models that may accelerate clinical adoption of RL for optimal decision-selection in adaptive radiotherapy.

The unmet potential of artificial intelligence in veterinary medicine

“…In this approach, the final model is achieved through the maximization of rewards, as opposed to minimizing a cost or loss function. 13,14 Unlike supervised ML methods, the data does not need to be labeled and the algorithm can be designed to generate labels for unlabeled data as an output. The algorithm attempts to solve a problem with different approaches and is rewarded (or punished) on the basis of the actions taken.…”

The role of artificial intelligence in veterinary radiation oncology

“…have demonstrated a technique for adaptive radiotherapy (ART) by using DVH‐guided IMRT optimization algorithm for automated treatment planning and found significant improvements by readjusting and optimizing the voxel weights 45 . Reinforcement machine learning has been applied to ART to estimate tumor control probability and normal tissue complication probability, as well as optimizing individual treatment dose distributions based on patients’ clinical, geometric, and physiological parameters 46,47 . Investigations have also been performed on determining which patients would benefit from ART, and when to best apply ART within a patient's treatment course as a result tumor shrinkage, organ movement, or changes in patient set up.…”

“…45 Reinforcement machine learning has been applied to ART to estimate tumor control probability and normal tissue complication probability, as well as optimizing individual treatment dose distributions based on patients' clinical, geometric, and physiological parameters. 46,47 Investigations have also been performed on determining which patients would benefit from ART, and when to best apply ART within a patient's treatment course as a result tumor shrinkage, organ movement, or changes in patient set up. Several commercial vendors of treatment delivery systems such as Accuracy's Radixact TM and Cyberknife TM (Accuray, Sunnyvale, CA, USA), Elekta's Unity MRlinac (Elekta, Atlanta, GA, USA), and Varian's Ethos (Varian, Palo Alto, CA, USA) offer various levels of ART integrated within the treatment delivery and planning chain, but again, the application of these algorithms in the clinical practice of veterinary medicine has not been well described in the literature (Figure 2).…”

A Gentle Introduction to Reinforcement Learning and its Application in Different Fields

“…Use of RL in cancer research is not new and have been used in cancer detection, classification of tumors and diagnosis. Many useful contribution of RL in the treatment of cancer in recent years include: ( [30], [31], [32], [33], [34], [35] and [36]). We have also the seen use of RL in the treatment of schizophrenia, anemia, and sepsis e.g.…”