Discordant bioinformatic predictions of antimicrobial resistance from whole-genome sequencing data of bacterial isolates: An inter-laboratory study

Abstract

BackgroundAntimicrobial resistance (AMR) poses a threat to public health. Clinical microbiology laboratories typically rely on culturing bacteria for antimicrobial susceptibility testing (AST). As the implementation costs and technical barriers fall, whole-genome sequencing (WGS) has emerged as a 'one-stop' test for epidemiological and predictive AST results. Few published comparisons exist for the myriad analytical pipelines used for predicting AMR. To address this, we performed an inter-laboratory study providing sets of participating researchers with identical short-read WGS data sequenced from clinical isolates, allowing us to assess the reproducibility of the bioinformatic prediction of AMR between participants and identify problem cases and factors that lead to discordant results. MethodsWe produced ten WGS datasets of varying quality from cultured carbapenem-resistant organisms obtained from clinical samples sequenced on either an Illumina NextSeq or HiSeq instrument. Nine participating teams ('participants') were provided these sequence data without any other contextual information. Each participant used their own pipeline to determine the species, the presence of resistance-associated genes, and to predict susceptibility or resistance to amikacin, gentamicin, ciprofloxacin and cefotaxime. ResultsIndividual participants predicted different numbers of AMR-associated genes and different gene variants from the same clinical samples. The quality of the sequence data, choice of bioinformatic pipeline and interpretation of the results all contributed to discordance between participants. Although much of the inaccurate gene variant annotation did not affect genotypic resistance predictions, we observed low specificity when compared to phenotypic AST results but this improved in samples with higher read depths. Had the results been used to predict AST and guide treatment a different antibiotic would have been recommended for each isolate by at least one participant. ConclusionsWe found that participants produced discordant predictions from identical WGS data. These challenges, at the final analytical stage of using WGS to predict AMR, suggest the need for refinements when using this technology in clinical settings. Comprehensive public resistance sequence databases and standardisation in the comparisons between genotype and resistance phenotypes will be fundamental before AST prediction using WGS can be successfully implemented in standard clinical microbiology laboratories.

Improved inference and prediction of bacterial genotype-phenotype associations using pangenome-spanning regressions

“…Using simulated data we demonstrate improved power and false-discovery rate at the single variant level compared with fixed and random effect models, and illustrate this use in practise on antibiotic resistance phenotypes in two species. We show further results which find similar accuracy between new machine-learning and simpler approaches, consistent with previous studies (4,5,24) . Additionally, our approach was able to estimate trait heritability without assuming specific effect size distributions, which are unproven in bacterial populations.…”

“…Batch differences such as genotyping methods between cohorts exaggerate this problem, so a consistent approach (such as the one we provide here) should be used. Unsurprisingly, curated resistance sets -the result of decades of research -still generally perform better, although even this in silico method loses accuracy between datasets (24) . Less well understood and potentially polygenic phenotypes such as virulence offer an attractive target for our model, as we demonstrated on two Streptococcal pathogens.…”

Beyond the pan-genome: current perspectives on the functional and practical outcomes of the distributed genome hypothesis

“…As with any system of analysis, appropriate technical requirements need to be instituted to prevent the caveats associated with poor quality sequencing that can lead to low-coverage and lost sequences. Doyle et al [ 146 ] have highlighted the need for quality sequence for clinical diagnostics wherein they showed missed WGS-based antibiotic resistance calls due to short read, low-coverage data. This has led to the push to adopt long-read, and even circular consensus sequencing, which obviate these issues [ 37 , 147–148 ].…”

Genomic diversity ofEscherichia coliisolates from healthy children in rural Gambia

“…IncFIB (AP001918) was the most common plasmid Inc type from our study, in line with the observation that IncF plasmids are frequently associated with the dissemination of resistance (49). However, a limitation of our study is that we did not perform phenotypic antimicrobial resistance testing, although Doyle et al (50) reported that only a small proportion of genotypic AMR predictions are discordant with phenotypic results.…”