Validation of a Bioinformatics Workflow for Routine Analysis of Whole-Genome Sequencing Data and Related Challenges for Pathogen Typing in a European National Reference Center: Neisseria meningitidis as a Proof-of-Concept

Abstract

Despite being a well-established research method, the use of whole-genome sequencing (WGS) for routine molecular typing and pathogen characterization remains a substantial challenge due to the required bioinformatics resources and/or expertise. Moreover, many national reference laboratories and centers, as well as other laboratories working under a quality system, require extensive validation to demonstrate that employed methods are “fit-for-purpose” and provide high-quality results. A harmonized framework with guidelines for the validation of WGS workflows does currently, however, not exist yet, despite several recent case studies highlighting the urgent need thereof. We present a validation strategy focusing specifically on the exhaustive characterization of the bioinformatics analysis of a WGS workflow designed to replace conventionally employed molecular typing methods for microbial isolates in a representative small-scale laboratory, using the pathogen Neisseria meningitidis as a proof-of-concept. We adapted several classically employed performance metrics specifically toward three different bioinformatics assays: resistance gene characterization (based on the ARG-ANNOT, ResFinder, CARD, and NDARO databases), several commonly employed typing schemas (including, among others, core genome multilocus sequence typing), and serogroup determination. We analyzed a core validation dataset of 67 well-characterized samples typed by means of classical genotypic and/or phenotypic methods that were sequenced in-house, allowing to evaluate repeatability, reproducibility, accuracy, precision, sensitivity, and specificity of the different bioinformatics assays. We also analyzed an extended validation dataset composed of publicly available WGS data for 64 samples by comparing results of the different bioinformatics assays against results obtained from commonly used bioinformatics tools. We demonstrate high performance, with values for all performance metrics >87%, >97%, and >90% for the resistance gene characterization, sequence typing, and serogroup determination assays, respectively, for both validation datasets. Our WGS workflow has been made publicly available as a “push-button” pipeline for Illumina data at https://galaxy.sciensano.be to showcase its implementation for non-profit and/or academic usage. Our validation strategy can be adapted to other WGS workflows for other pathogens of interest and demonstrates the added value and feasibility of employing WGS with the aim of being integrated into routine use in an applied public health setting.

Isolation of Drug-Resistant Gallibacterium anatis from Calves with Unresponsive Bronchopneumonia, Belgium

“…We performed genotypic resistance gene detection, as described by Bogaerts et al (23), against the Res-Finder database (24). We defined AMR gene clusters as resistance genes on the same contig within a sample.…”

Next-Generation Sequencing: An Eye-Opener for the Surveillance of Antiviral Resistance in Influenza

“…Although, there are many tools available to analyze NGS results, many of these require substantial bioinformatic expertise because they are only available using the command line on Linux. Therefore, the spread of more web-based platforms with a user-friendly interface within the scientific community would be an advance in the use of WGS for non-bioinformatic experts [60].…”

Detailed Evaluation of Data Analysis Tools for Subtyping of Bacterial Isolates Based on Whole Genome Sequencing: Neisseria meningitidis as a Proof of Concept

“…Classical subtyping data were provided by the NRC and determined as described by Bertrand et al (2011). Designation of antigen gene alleles for porA, porB, and fetA genes were obtained with the Neisseria pipeline described by Bogaerts et al (2019), by comparing assembled sequencing data to reference alleles downloaded from the Neisseria PubMLST 1 database (Jolley et al, 2018).…”

“…ND, not determined; NT, non-serotypeable; NST, non-serosubtypeable; NE, clonal complex does not exist. Clonal complex (cc), sequence type (ST), PorB VR, PorA VR1, PorA VR2, and FetA determined in silico as described in Bogaerts et al (2019), Serogroup:serotype:serosubtype determined in vitro as described in Bertrand et al (2011). In Bertrand et al (2011), isolate 2008-114 was identified as ST2592.…”

“…Prior to the analysis, a number of different parameter sets have been briefly evaluated for some workflows (e.g., the fragmentation size and the percentage identity cut-off for Panseq, and the minimal coverage for LyveSET) in order to choose the optimal configuration for the current study. 3 https://github.com/phe-bioinformatics/kmerid cgMLST The tested implementation of the cgMLST approach is described in Bogaerts et al (2019). It is based on the regularly updated cgMLST typing scheme maintained by the PubMLST platform (Jolley et al, 2018).…”