A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images

Abstract

In the automatic diagnosis of ocular toxoplasmosis (OT), Deep Learning (DL) has arisen as a powerful and promising approach for diagnosis. However, despite the good performance of the models, decision rules should be interpretable to elicit trust from the medical community. Therefore, the development of an evaluation methodology to assess DL models based on interpretability methods is a challenging task that is necessary to extend the use of AI among clinicians. In this work, we propose a novel methodology to quantify the similarity between the decision rules used by a DL model and an ophthalmologist, based on the assumption that doctors are more likely to trust a prediction that was based on decision rules they can understand. Given an eye fundus image with OT, the proposed methodology compares the segmentation mask of OT lesions labeled by an ophthalmologist with the attribution matrix produced by interpretability methods. Furthermore, an open dataset that includes the eye fundus images and the segmentation masks is shared with the community. The proposal was tested on three different DL architectures. The results suggest that complex models tend to perform worse in terms of likelihood to be trusted while achieving better results in sensitivity and specificity.

Benchmarking Deep Learning Frameworks for Automated Diagnosis of Ocular Toxoplasmosis: A Comprehensive Approach to Classification and Segmentation

“…This improved accuracy is likely a result of our superior preprocessing methods. This inference gains further support from our improved accuracy of 0.976 with the VGG16 model, which is 0.07 higher than the results noted in reference [43].…”

“…The results indicates that our models demonstrate higher performance over the previous research. Notably, even our least effective model, ResNet50, achieves an accuracy of 0.973, surpassing the former state-of-the-art (SOTA) model VGG16 [43]. This improved accuracy is likely a result of our superior preprocessing methods.…”

“…A comprehensive overview of previous work has been presented in Table 1. In the recent studies [42] and [43], datasets from Hospital de Clínicas and Hospital General Pedriático Acosta Ñu medical centers in Asunción, Paraguay [44] were utilized for the diagnosis of OT. The former study employed a pre-trained RestNet18 network, while the latter used a trust-based method, achieving accuracies of 0.938 and 0.969, respectively.…”