Diagnostic Accuracy of Artificial Intelligence Algorithm incorporated into MobileODT Enhanced Visual Assessment for triaging Screen Positive Women after Cervical Cancer Screening

Abstract

Introduction : The goal of cervical cancer screening is to detect precancerous precursor lesions that can be treated in the preinvasive stage. Colposcopy is important for triaging of any abnormal cervical screening test. Scarcity of trained Colposcopists and colposcopy centres is a big hurdle to screening programs in lower and middle income countries. Objectives of the study: The objective was to assess the performance of the Artificial Intelligence based incorporated into the MobileODT Enhanced Visual Assessment (EVA Visual Check) against physician colposcopic diagnosis and the gold standard of histopathology. Material and Methods: A cross sectional observational study was conducted on women referred to our colposcopy clinic following an abnormal screening test. Colposcopic examination was performed by trained physicians using the mobile optical device technologies (ODT) Enhanced Visual Assessment (EVA) system and the images were saved. The physician colposcopic impression was recorded, biopsy performed if any acetowhite lesions were found. Images taken with the EVA system were uploaded on a cloud and analysed by the Visual Check algorithm. Correlation of Physician impression using the mobile ODT colposcope was compared with the Visual Check analysis. The sensitivity, specificity, positive predictive value, negative predictive value of physician’s impression and Visual Check analysis was compared with the final histopathological analysis or cytology . Cases with normal cytology and normal colposcopy did not undergo biopsy and these were considered normal. Result: A total of 2050 women were screened from January 2021 to September 2021.We studied 147 women, EVA Visual check had a Sensitivity of 86.8% (75-95), Specificity 28.7% (20-39), Positive Predictive Value of 40.7% (32-50) and negative predictive value of 79.4% (62-91) and diagnostic accuracy of 49.7% (41-58) for CIN 1+ lesions. EVA visual check has sensitivity 89.3% (72-98),specificity 26.1% (18-35), PPV 22.1% (15-31), NPV 91.2% (76-98) and diagnostic accuracy of 38.1% (30-46) for CIN 2 + lesions. Physician impression had a sensitivity of 86.8% (75-95), specificity of 81.9% (73-89) positive predictive value(PPV) 73.0% (60-83) negative predictive value (NPV) 91.7% (84-97) and diagnostic accuracy 83.7% (77-89) as compared to histopathological diagnosis of CIN1+ lesions and sensitivity of 92.9% (76-99) ,specificity of 68.9% ( 60-77%) ,positive predictive value (PPV) 41.3% (29-54) ,negative predictive value (NPV) 97.6% ( 92-100) and diagnostic accuracy 73.5% (66-80) as compared to histopathological diagnosis of CIN 2 + lesions as the gold standard Conclusion: Mobile ODT EVA colposcope with AI has sensitivity comparable to physician’s diagnosis while specificity ,PPV and NPV was less than that of physician diagnosis. It is valuable for triage of screen positive women for further management.

Deep Learning Diagnostic Classification of Cervical Images to Augment Colposcopic Impression

“…12 The National Cancer Institute in the United States spearheads the development of models for use in low-middle income countries with promising initial results, 23 efforts to converge toward a usable model, 24 and a program for developing a novel screening/ treatment solution for resource-limited settings. 25,26 The same group worked with images from a mobile phone-based colposcope, demonstrating some potential in learning to reproduce expert reviewer predictions 27 ; however, a recent report from its implementation suggests that its performance was subpar to the colposcopists, 28 highlighting the gap between model development and deployment.…”

Reproducible And Clinically Translatable Deep Neural Networks For Cervical Screening

“…Our work builds on prior work highlighting improvements in repeatability of model predictions made by certain design choices (36,37). Our work also stands out among the paucity of current approaches that have utilized AI and DL for cervical screening (21)(22)(23)(24); as aforementioned, these are largely plagued by overfitting and no consideration of repeatability. The dearth of work investigating repeatability of AI models designed for clinical translation in the current DL and medical image classification literature has meant that no rigorous study, to the best of our knowledge, has employed repeatability as a model selection criterion.…”

“…Subsequent design choices of our work are optimized to improve clinical translatability. Prior work (21)(22)(23)(24) has shown us that while binary classifiers for cervical image-based cervical precancer+ detection can achieve competitive performance in a given internal seed dataset, they translate poorly when tested in different settings; uncertain cases can be misclassified, and predictions tend to oscillate between the two classes. This oscillation phenomenon could prevent a precancer+ woman from accessing further evaluation (i.e., false negative) or direct a normal woman through unnecessary, potentially invasive tests (i.e., false positive).…”

Reproducible And Clinically Translatable Deep Neural Networks For Cervical Screening

“…Our clinical intuition from working with binary models as well as prior empirical work have informed us that these models frequently fail to capture the inherent uncertainty with ambiguous samples (21)(22)(23)(24). These uncertain samples are of two intersecting kinds: samples that are uncertain to the clinician ("rater uncertainty") and samples that are uncertain to the model i.e., where the model reports low confidence scores ("model uncertainty"); both instances can lead to incorrect classification and subsequent misinformed downstream actions for these patients.…”

“…This leads to apparent initial promise, with either poor performance on or absence of held-aside test sets for evaluating true model performance. When deployed in different settings, these models fail to return consistent scores and accurately detect precancers (21)(22)(23)(24). This poses significant concerns when considering downstream deployment in various LMIC, where model predictions directly inform the course of treatment, and where screening opportunities are limited.…”