Evaluation of parameters affecting performance and reliability of machine learning-based antibiotic susceptibility testing from whole genome sequencing data

Abstract

20Prediction of antibiotic resistance phenotypes from whole genome sequencing data by 21 machine learning methods has been proposed as a promising platform for the 22 development of sequence-based diagnostics. However, there has been no systematic 23 evaluation of factors that may influence performance of such models, how they might 24 apply to and vary across clinical populations, and what the implications might be in the 25 clinical setting. Here, we performed a meta-analysis of seven large Neisseria 26 gonorrhoeae datasets, as well as Klebsiella pneumoniae and Acinetobacter baumannii 27 datasets, with whole genome sequence data and antibiotic susceptibility phenotypes 28 using set covering machine classification, random forest classification, and random forest 29 regression models to predict resistance phenotypes from genotype. We demonstrate how 30 model performance varies by drug, dataset, resistance metric, and species, reflecting the 31 complexities of generating clinically relevant conclusions from machine learning-derived 32 models. Our findings underscore the importance of incorporating relevant biological and 33 epidemiological knowledge into model design and assessment and suggest that doing so 34 can inform tailored modeling for individual drugs, pathogens, and clinical populations. We 35 further suggest that continued comprehensive sampling and incorporation of up-to-date 36 whole genome sequence data, resistance phenotypes, and treatment outcome data into 37 model training will be crucial to the clinical utility and sustainability of machine learning-38 based molecular diagnostics. 39 40 Author Summary: 41 3 Machine learning-based prediction of antibiotic resistance from bacterial genome 42 sequences represents a promising tool to rapidly determine the antibiotic susceptibility 43 profile of clinical isolates and reduce the morbidity and mortality resulting from 44 inappropriate and ineffective treatment. However, while there has been much focus on 45 demonstrating the diagnostic potential of these modeling approaches, there has been 46 little assessment of potential caveats and prerequisites associated with implementing 47 predictive models of drug resistance in the clinical setting. Our results highlight significant 48 biological and technical challenges facing the application of machine learning-based 49 prediction of antibiotic resistance as a diagnostic tool. By outlining specific factors 50 affecting model performance, our findings provide a framework for future work on 51 modeling drug resistance and underscore the necessity of continued comprehensive 52 sampling and reporting of treatment outcome data for building reliable and sustainable 53 diagnostics. 54 55 56 57 58 4 Introduction: 59

Adaptation to the cervical environment is associated with increased antibiotic susceptibility in Neisseria gonorrhoeae

“…The majority (59.4%) of these isolates had MICs that were lower than expected, indicative of increased susceptibility unexplained by the genetic determinants in our model. Overall MIC variance explained by known resistance mutations was relatively low (adjusted R 2 = 0.667), in agreement with prior studies that employed whole-genome supervised learning algorithms to predict azithromycin resistance 13 . MIC variance explained by known resistance mutations was also low for ceftriaxone (adjusted R 2 = 0.674) but higher for ciprofloxacin (adjusted R 2 = 0.937), with 2.02% and 2.90% of strains, respectively, exhibiting two dilutions or lower reported MICs compared to predictions, similarly indicating unexplained susceptibility.…”

Predicting antimicrobial resistance using conserved genes

“…The average very major error rate (VME), which is defined as resistant genomes that are erroneously predicted to be susceptible, and the average major error rate (ME), which is defined as susceptible genomes that are erroneously predicted to be resistant, tend to go down as gene set size increases. Although the core gene set models described in Fig 1 have lower F1 scores and higher error rates than full-genome models that have been published previously [ 21 – 24 , 27 , 32 ], their accuracies are striking given the small sizes of the input data sets and the removal of well-annotated AMR genes.…”

Interpreting k-mer–based signatures for antibiotic resistance prediction

“…Following Hicks et al. [ 28 ], we refer to the average of the (optimal) sensitivity and specificity as balanced accuracy (bACC). Finally, we selected the sparsest model that allowed maximization of the bACC up to 1 point, in order to reduce the risk of overfitting.…”

“…Second, regarding the type of ML algorithms, boosting algorithms [ 4 , 8 , 21 ], penalized regression models [ 10 , 17 , 23 ], decision trees [ 26 ], random forest [ 10 , 27 ], neural networks [ 17 ], and set cover machines [ 22 , 26 ] have already been successfully deployed in this context. While each algorithm has its own merits and shortcomings, several studies reported comparable global performance for various algorithms, with specific variations by drug and microbial species [ 10 , 17 , 28 ]. Finally, different kinds of antibiotic susceptibility information can be considered: either discrete when the objective is to distinguish susceptible from resistant (or non-susceptible) ones [ 10 , 17 , 21 , 22 ], or continuous, where one seeks to predict the minimum inhibitory concentration (MIC) of the antimicrobial agent itself [ 3 , 4 , 8 ].…”