Abstract S6-07: Double blinded validation study to assess performance of IBM artificial intelligence platform, Watson for oncology in comparison with Manipal multidisciplinary tumour board – First study of 638 breast cancer cases

Abstract

Background: IBM Watson for Oncology (WFO) developed in collaboration with Memorial Sloan Kettering Cancer Centre is a cognitive computing system able to extract structured data from free text documents using natural language processing (NLP). It is a technology platform that uses NLP and machine learning to reveal insights from large amounts of unstructured data. Currently WFO provides treatment options only for breast, lung and colorectal cancers. In the present study we try to evaluate concordance of WFO treatment recommendations with Manipal multidisciplinary tumor board (MMDT),a quaternary health care centre for 638 breast cancer cases. Materials and Methods: MMDT treatment recommendation and relevant clinic-pathological data of 638 breast cancer cases both localised (514) and metastatic (124) disease which were treated in last 3 years was collected and the collected data of all the patients was entered in WFO. The treatment recommendations & the time required to enter the data and the time taken by WFO to give recommendations after analyzing the data was recorded. Treatment recommendation by WFO came in three categories with colour coding green, orange and red . Green is the RECommended standard treatment (REC), Orange is For Consideration (FC) and Red is Not RECommended (NREC). Concordance between MMDT and WFO was analysed for all cases as per stage and receptor status. Results: Of the treatment recommendations given by MMDT, WFO provided 46.4% in REC, 26.1% in FC, 21.5% in NREC. Nearly 73% of the MMDT treatment recommendations were in WFO REC and WFO-FC group. However 6% of the treatment provided by MMDT was not available with WFO. Treatment recommendations from WFO were concordant with MMDT in nearly 80% of time in non-metastatic and 45% in metastatic disease. In subset analysis of breast cancer with respect to receptor status the WFO-REC treatment was high with triple negative disease with 67.9% and least with Her 2neu negative disease with 35%. Across all subsets with respect to receptor status concordance was better with non-metastatic disease over metastatic disease which was statistically significant. The mean time taken to collect the data and enter was 20 minutes which gradually decreased with acquaintance after 10 cases to 12 minutes. Metastatic disease took longer time (5-7 minutes more) over localized disease in all the groups. WFO took a median of 40 seconds to capture, analyze and give the treatment recommendations. Conclusions: WFO-REC and WFO-FC together were in 73 % of time concordant with the MMDT treatment recommendations. However with respect to metastatic disease and harmone positive Her 2 neu –ve disease there still needs to be lots of improvement from WFO. WFO is a step towards personalized medicine. It should be kept in mind that WFO will be only an assisting tool and it will never be able to replace the patient-doctor relationship which is a very essential component of treating a patient suffering with cancer. WFO will be a reliable artificial intelligence tool for every cancer center and its multidisciplinary tumor board and will change the quality of care in oncology. Citation Format: Somashekhar SP, Kumarc R, Rauthan A, Arun KR, Patil P, Ramya YE. Double blinded validation study to assess performance of IBM artificial intelligence platform, Watson for oncology in comparison with Manipal multidisciplinary tumour board – First study of 638 breast cancer cases [abstract]. In: Proceedings of the 2016 San Antonio Breast Cancer Symposium; 2016 Dec 6-10; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2017;77(4 Suppl):Abstract nr S6-07.

Artificial intelligence in healthcare: past, present and future

“…Cancer: Somashekhar et al demonstrated that the IBM Watson for oncology would be a reliable AI system for assisting the diagnosis of cancer through a double-blinded validation study. 19 Esteva et al analysed clinical images to identify skin cancer subtypes. 20 …”

Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness

“…However, published works describing WFO do not report relevant statistical (eg, discrimination, calibration) and clinically oriented (eg, net benefit type) performance metrics. Instead, they focus on concordance (true positive rate where the ground truth is provided by physician—that is, the proportion of instances where WFO’s recommendation agrees with that of the treating physician363738). We recommend the following guidance for researchers to avoid such pitfalls:…”

Application of Artificial Intelligence in Modern Healthcare System

“…Farina et al experienced the control of the offline man-machine edge, which applies the release timings for the spinal motor neurons for controlling the prosthesis of the upper limb. IBM Watson for the oncology diagnosis can be a consistent AI for cancer diagnosis from start to the end, which was explained by Somashekhar et al [46] by a double-blinded validation study. A clinical image was examined for recognizing skin cancer subtypes [20].…”