Best practices for benchmarking germline small-variant calls in human genomes

Abstract

Standardized benchmarking methods and tools are essential to robust accuracy assessment of NGS variant calling. Benchmarking variant calls requires careful attention to definitions of performance metrics, sophisticated comparison approaches, and stratification by variant type and genome context. To address these needs, the Global Alliance for Genomics and Health (GA4GH) Benchmarking Team convened representatives from sequencing technology developers, government agencies, academic bioinformatics researchers, clinical laboratories, and commercial technology and bioinformatics developers for whom benchmarking variant calls is essential to their work. This team addressed challenges in (1) matching variant calls with different representations, (2) defining standard performance metrics, (3) enabling stratification of performance by variant type and genome context, and (4) developing and describing limitations of high-confidence calls and regions that can be used as “truth”. Our methods are publicly available on GitHub (https://github.com/ga4gh/benchmarking-tools) and in a web-based app on precisionFDA, which allow users to compare their variant calls against truth sets and to obtain a standardized report on their variant calling performance. Our methods have been piloted in the precisionFDA variant calling challenges to identify the best-in-class variant calling methods within high-confidence regions. Finally, we recommend a set of best practices for using our tools and critically evaluating the results.

Clair: Exploring the limit of using a deep neural network on pileup data for germline variant calling

“…Fast and accurate variant calling is essential for both research and clinical applications of human genome sequencing 1,2 . Algorithms, best practices and benchmarking guidelines have been established for how to use Illumina sequencing to call germline small variants, including single-nucleotide polymorphisms (SNPs) and insertions/deletions (indels) 3–6 . In recent years, single-molecule sequencing (SMS) technologies have emerged for a variety of important applications 7 .…”

Accuracy and efficiency of germline variant calling pipelines for human genome data

“…The construction of the truth set, and strengths and weaknesses based on variant type and genome context should be considered. The GiaB benchmark sets were built from the consensus of multiple variant callers on Illumina short-read sequencing with the aid of a pedigree analysis, integration of structural variants identified with long fragment technologies by PacBio and 10X Genomics, and HuRef genome analysis using Sanger sequencing 35 . Nearly all the “true” variants in NA12878 sample are present in the resource files (e.g.…”

“…However, the “synthetic-diploid” call set currently contains some errors that were intrinsically present in the long reads 27 . It is thus recommended to use a less strict benchmarking strategy (“local matches” method) for comparisons 27,35 . Here, the evaluation using “genotype match” as it applied in NA12878 datasets was performed as well (Table S4).…”

“…The variant callings of WGS data from the mixture of CHM1 and CHM13 were compared to the “synthetic-diploid” benchmark truth callset and high-confidence regions using the CHM-eval kit tool (version 20180222, vcfeval comparison engine) 27 . The definitions of true positive (TP), false positive (FP) and false negative (FN) were based on the types of variant matching stringencies “genotype match” (most strict) and “local match” (less strict) 35 . Precision, Recall and F1-score were calculated as TP/(TP+FP), TP/(TP+FN) and 2*TP/(2*TP+FN+FP), respectively.…”

Genomic loci susceptible to systematic sequencing bias in clinical whole genomes

“…However, this top-down approach cannot identify which variant calls are most likely when there is disagreement between the results from different sequencing protocols. In addition, at loci that have the same sequencing biases across all or most sequencing technologies, assuming that the consensus call is true can lead to errors being falsely called as real high-confidence variants (Krusche et al 2019). Another drawback to this is that clinically collected samples can vary in quality and contamination and may introduce variants with low allelic fractions not seen in reference genomes.…”