Comparison of an oncology clinical decision-support system’s recommendations with actual treatment decisions

Abstract

Objective IBM(R) Watson for Oncology (WfO) is a clinical decision-support system (CDSS) that provides evidence-informed therapeutic options to cancer-treating clinicians. A panel of experienced oncologists compared CDSS treatment options to treatment decisions made by clinicians to characterize the quality of CDSS therapeutic options and decisions made in practice. Methods This study included patients treated between 1/2017 and 7/2018 for breast, colon, lung, and rectal cancers at Bumrungrad International Hospital (BIH), Thailand. Treatments selected by clinicians were paired with therapeutic options presented by the CDSS and coded to mask the origin of options presented. The panel rated the acceptability of each treatment in the pair by consensus, with acceptability defined as compliant with BIH’s institutional practices. Descriptive statistics characterized the study population and treatment-decision evaluations by cancer type and stage. Results Nearly 60% (187) of 313 treatment pairs for breast, lung, colon, and rectal cancers were identical or equally acceptable, with 70% (219) of WfO therapeutic options identical to, or acceptable alternatives to, BIH therapy. In 30% of cases (94), 1 or both treatment options were rated as unacceptable. Of 32 cases where both WfO and BIH options were acceptable, WfO was preferred in 18 cases and BIH in 14 cases. Colorectal cancers exhibited the highest proportion of identical or equally acceptable treatments; stage IV cancers demonstrated the lowest. Conclusion This study demonstrates that a system designed in the US to support, rather than replace, cancer-treating clinicians provides therapeutic options which are generally consistent with recommendations from oncologists outside the US.

A comparison between clinical decision support system and clinicians in breast cancer

“…It would be 62% if only the ‘recommended’ option was included. To further the application of WFO in China, we also started a phase III trial to compare the concordance between clinicians' decisions and their decisions with WFO [ [18] , [19] , [20] ]. The adherence of clinicians' decision to National Comprehensive Cancer Network guidelines was 82%, however, it was only 56% when compared with CSCO guidelines.…”

Artificial intelligence in clinical decision support systems for oncology

“…It should be emphasized that the CDSS cannot replace oncologists at present 74 , 75 . It should be positioned as a good assistant or teacher for young doctors 13 .…”

“…What's more, colorectal cancers exhibited the highest proportion of identical or equally acceptable treatments; however, stage IV cancers demonstrate the lowest. Therefore, collaborative intelligence already exists and very likely to dominate in the future, the CDSS that could support, rather than replace the clinicians provides therapeutic options which are generally consistent with recommendations 74 . Also, medicine is not only a science but it also involves other aspects, such as social and psychological factors 77 , 78 .…”

Physicians’ Perceptions of and Satisfaction With Artificial Intelligence in Cancer Treatment: A Clinical Decision Support System Experience and Implications for Low-Middle–Income Countries

“…WfO concordance rates varied widely across countries for many reasons, including differences in standard treatment guidelines, resource availability, and physician or patient preferences [15]. It is well recognized that concordance studies do not measure system accuracy but instead assess agreement with decisions made in practice, which may or may not reflect evidence-based decisions [16].…”