Patient-level performance evaluation of a smartphone-based malaria diagnostic application

Abstract

Background Microscopic examination is commonly used for malaria diagnosis in the field. However, the lack of well-trained microscopists in malaria-endemic areas impacted the most by the disease is a severe problem. Besides, the examination process is time-consuming and prone to human error. Automated diagnostic systems based on machine learning offer great potential to overcome these problems. This study aims to evaluate Malaria Screener, a smartphone-based application for malaria diagnosis. Methods A total of 190 patients were recruited at two sites in rural areas near Khartoum, Sudan. The Malaria Screener mobile application was deployed to screen Giemsa-stained blood smears. Both expert microscopy and nested PCR were performed to use as reference standards. First, Malaria Screener was evaluated using the two reference standards. Then, during post-study experiments, the evaluation was repeated for a newly developed algorithm, PlasmodiumVF-Net. Results Malaria Screener reached 74.1% (95% CI 63.5–83.0) accuracy in detecting Plasmodium falciparum malaria using expert microscopy as the reference after a threshold calibration. It reached 71.8% (95% CI 61.0–81.0) accuracy when compared with PCR. The achieved accuracies meet the WHO Level 3 requirement for parasite detection. The processing time for each smear varies from 5 to 15 min, depending on the concentration of white blood cells (WBCs). In the post-study experiment, Malaria Screener reached 91.8% (95% CI 83.8–96.6) accuracy when patient-level results were calculated with a different method. This accuracy meets the WHO Level 1 requirement for parasite detection. In addition, PlasmodiumVF-Net, a newly developed algorithm, reached 83.1% (95% CI 77.0–88.1) accuracy when compared with expert microscopy and 81.0% (95% CI 74.6–86.3) accuracy when compared with PCR, reaching the WHO Level 2 requirement for detecting both Plasmodium falciparum and Plasmodium vivax malaria, without using the testing sites data for training or calibration. Results reported for both Malaria Screener and PlasmodiumVF-Net used thick smears for diagnosis. In this paper, both systems were not assessed in species identification and parasite counting, which are still under development. Conclusion Malaria Screener showed the potential to be deployed in resource-limited areas to facilitate routine malaria screening. It is the first smartphone-based system for malaria diagnosis evaluated on the patient-level in a natural field environment. Thus, the results in the field reported here can serve as a reference for future studies.

Intelligent preliminary diagnosis system for diseases with similar clinical presentation

“…The study's results show that the system, created with Python and PyQt5, achieved a precision of 95%, a sensitivity of 90%, and a specificity of 93% when assessing 35 clinical cases involving seven different conditions with similar initial symptoms. Comparing these findings with past research, it is clear that the proposed system excels in terms of accuracy when compared to similar tools like the Malaria Screener [11], which measured 74.1% accuracy in detecting Plasmodium falciparum. Additionally, the proposed system surpasses other tools that have utilized innovative methods, such as logical criteria fusion and variable evocation strength [12], which averaged 75% precision for initial diagnosis.…”

“…This greatly expands the range of applicability, as it is versatile and adaptable to a wide variety of medical conditions. This differs from other studies that focus on specific diseases such as malaria [11], COVID-19 [13], influenza [17], or chikungunya [15]. The developed system is notable for being lightweight and compatible with the Windows operating system.…”

“…Yu et al [11] used conventional microscopy to diagnose malaria in endemic areas but noted the lack of trained microscopists and the propensity for errors. In response, they evaluated Malaria Screener, a smartphone application, in a study in rural Sudan.…”

Edge Artificial Intelligence (AI) for real-time automatic quantification of filariasis in mobile microscopy

“…For example, Yu et al evaluated a malaria screener employing a custom CNN to detect Plasmodium falciparum using a smartphone on both thin and thick blood smears. Developed with 150 patients and 50 healthy subjects, the model achieved an accuracy of 74.1% compared to expert microscopy on a test set of 190 patients, meeting the WHO level 3 requirement for parasite detection [ 43 , 52 , 53 ]. Armstrong et al proposed an object detection algorithm using ResNet101 to detect Schistosoma eggs on Google Pixel 4.…”

Evaluating the performance of hyperparameters for unbiased and fair machine learning

“…22,23 In particular, the models were trained and evaluated using a dataset of 364 patient cases acquired in Bangladesh, comprising 3,532 images, 7,952 instances of P. falciparum, 4,346 instances of P. vivax, and over 860,000 instances of uninfected cells. Another dataset with 190 patients was acquired in Sudan, 24 comprising 874 images, 336 instances of P. falciparum, 59 instances of P. vivax, and over 220000 instances of uninfected cells. The YOLOv8x model was initialized using COCO weights.…”