Artificial Intelligence-Based Differential Diagnosis: Development and Validation of a Probabilistic Model to Address Lack of Large-Scale Clinical Datasets (Preprint)

Abstract

BACKGROUND Machine-learning or deep-learning algorithms for clinical diagnosis are inherently dependent on the availability of large-scale clinical datasets. Lack of such datasets and inherent problems such as overfitting often necessitate the development of innovative solutions. Probabilistic modeling closely mimics the rationale behind clinical diagnosis and represents a unique solution. OBJECTIVE The aim of this study was to develop and validate a probabilistic model for differential diagnosis in different medical domains. METHODS Numerical values of symptom-disease associations were utilized to mathematically represent medical domain knowledge. These values served as the core engine for the probabilistic model. For the given set of symptoms, the model was utilized to produce a ranked list of differential diagnoses, which was compared to the differential diagnosis constructed by a physician in a consult. Practicing medical specialists were integral in the development and validation of this model. Clinical vignettes (patient case studies) were utilized to compare the accuracy of doctors and the model against the assumed gold standard. The accuracy analysis was carried out over the following metrics: top 3 accuracy, precision, and recall. RESULTS The model demonstrated a statistically significant improvement (<i>P</i>=.002) in diagnostic accuracy (85%) as compared to the doctors’ performance (67%). This advantage was retained across all three categories of clinical vignettes: 100% vs 82% (<i>P</i><.001) for highly specific disease presentation, 83% vs 65% for moderately specific disease presentation (<i>P</i>=.005), and 72% vs 49% (<i>P</i><.001) for nonspecific disease presentation. The model performed slightly better than the doctors’ average in precision (62% vs 60%, <i>P</i>=.43) but there was no improvement with respect to recall (53% vs 56%, <i>P</i>=.27). However, neither difference was statistically significant. CONCLUSIONS The present study demonstrates a drastic improvement over previously reported results that can be attributed to the development of a stable probabilistic framework utilizing symptom-disease associations to mathematically represent medical domain knowledge. The current iteration relies on static, manually curated values for calculating the degree of association. Shifting to real-world data–derived values represents the next step in model development.

A Primer into the Current State of Artificial Intelligence in Gastroenterology

“…J Gastrointestin Liver Dis, June 2022 Vol. 31 No 2: 244-253 Support vector machine is a simpler type of algorithm, where each observation (e.g. patient) is drawn on a chart, using it's features as distance from the axis (e.g., patients with alcoholic hepatitis might have prothrombin time and bilirubin as features), and then a hyperplane (which is essentially a border between the two categories -e.g., severe and non-severe hepatitis) is drawn in order to separate the two categories of patients.…”

“…Artificial intelligence algorithms are very dependent on high quality data, much more so than the current practice. For an algorithm to correctly train itself and become proficient, it requires large amounts of data, and the data needs to be accurate and consistent [31]. This presents unique challenges, as the requirement for large amounts of data would suggest multi-center AI as a solution, but issues such as differing testing methodologies or test types between the various centers might impede the algorithms' ability to learn [32].…”