Objective IBM(R) Watson for Oncology (WfO) is a clinical decision-support system (CDSS) that provides evidence-informed therapeutic options to cancer-treating clinicians. A panel of experienced oncologists compared CDSS treatment options to treatment decisions made by clinicians to characterize the quality of CDSS therapeutic options and decisions made in practice. Methods This study included patients treated between 1/2017 and 7/2018 for breast, colon, lung, and rectal cancers at Bumrungrad International Hospital (BIH), Thailand. Treatments selected by clinicians were paired with therapeutic options presented by the CDSS and coded to mask the origin of options presented. The panel rated the acceptability of each treatment in the pair by consensus, with acceptability defined as compliant with BIH’s institutional practices. Descriptive statistics characterized the study population and treatment-decision evaluations by cancer type and stage. Results Nearly 60% (187) of 313 treatment pairs for breast, lung, colon, and rectal cancers were identical or equally acceptable, with 70% (219) of WfO therapeutic options identical to, or acceptable alternatives to, BIH therapy. In 30% of cases (94), 1 or both treatment options were rated as unacceptable. Of 32 cases where both WfO and BIH options were acceptable, WfO was preferred in 18 cases and BIH in 14 cases. Colorectal cancers exhibited the highest proportion of identical or equally acceptable treatments; stage IV cancers demonstrated the lowest. Conclusion This study demonstrates that a system designed in the US to support, rather than replace, cancer-treating clinicians provides therapeutic options which are generally consistent with recommendations from oncologists outside the US.

No abstract

6553 Background: Clinical decision-support systems (CDSS) such as Watson for Oncology (WFO) may reduce treatment variation in oncology, provided options offered by the system are at least as acceptable as expert, evidence-based options. Deviation from expert consensus in practice is not well documented. In this blinded study, WFO therapeutic options and treatment decisions made by individual oncologists at Bumrungrad International Hospital (BIH) were evaluated by expert panel. Methods: Treatments selected by BIH that were labeled as either “for consideration” or “not recommended” by WFO were evaluated by a panel of 3 oncologists in 2018. The panel evaluated WFO options and previous BIH treatments for prospective cases from 2016-2018, blinded to the source of treatment option. Consensus of panel rated treatment pairs as: identical; both acceptable and roughly equivalent; both acceptable, but one preferred; one is acceptable and the other, unacceptable; neither is acceptable. The results of 321 treatment choices for breast, lung, colon and rectal cancers were analyzed, and McNemar’s test, a modified pairwise chi-square, was applied to identify differences between BIH and WFO. Results: 71% of both BIH and WFO treatments across all 4 cancer types were considered acceptable or identical by the panel. In 18 cases (5.6%), WFO treatments were preferred; in 14 cases (4.4%), BIH cases were preferred. Unacceptable treatments by either BIH or WFO were identified in 15% and 23% of treatments, respectively. Statistical analysis of treatment pairs revealed no significant difference between BIH and WFO treatments for breast, colon and rectal cancer. Treatment for lung cancer differed significantly ( p = 0.004); in 6% of cases, WFO was unacceptable and BIH acceptable; in 1% of cases, BIH was unacceptable and WfO was acceptable. Conclusions: This study is one of the first to compare therapeutic options from CDSS to treatment decisions made in practice, evaluated in a blinded fashion by an expert panel. 71% of treatments suggested by WFO CDSS were as acceptable as those selected by clinicians at the point of care, and some were considered superior. Decisions made in practice were unacceptable to the panel in 15% of cases, suggesting a role for CDSS.

95 Background: Watson for Oncology (WFO) is an artificial intelligence (AI) based clinical decision-support tool trained by Memorial Sloan Kettering. This retrospective observational study of breast, lung, colon and rectal cancer examined the concordance of treatment options provided by WFO to treatments selected by clinicians at Bumrungrad International Hospital (BIH) as a function of stage or cancer type. Methods: Concordance between WFO treatment options and treatments selected by BIH clinicians (WFO-BIH concordance) was defined as identical or equally acceptable treatments, as determined by a panel of experts blinded to the source of treatment. Relationships between stage or type of cancer and WFO-BIH concordant treatments were evaluated by Chi-squared analysis. Results: Analysis revealed a statistically significant association ( P = 0.02) between cancer stage and concordance. For all 4 cancer types combined, stages I-III demonstrated higher concordance than stage IV. A highly significant association ( P < 0.001) between concordance and cancer type was identified. Colon cancer demonstrated the highest concordance, followed by rectal, lung and breast cancer. Reasons for discordance, when given, related to oncologist or patient preferences, and treatment availability. Conclusions: BIH clinicians tended to agree more with WFO therapeutic options for stage I-III cancers and colon cancer in general, as compared to relatively less agreement for stage IV cancers and breast cancer in general, suggesting the need to understand reasons for discordance among all cancer types and stages. An AI tool, trained by experts in the U.S., provides treatment options consistent with some therapies selected in international settings, but preferences and treatment availability may affect choices made in practice. [Table: see text]