Verification and validation of bioinformatics software without a gold standard: a case study of BWA and Bowtie

Abstract

BackgroundBioinformatics software quality assurance is essential in genomic medicine. Systematic verification and validation of bioinformatics software is difficult because it is often not possible to obtain a realistic "gold standard" for systematic evaluation. Here we apply a technique that originates from the software testing literature, namely Metamorphic Testing (MT), to systematically test three widely used short-read sequence alignment programs.ResultsMT alleviates the problems associated with the lack of gold standard by checking that the results from multiple executions of a program satisfy a set of expected or desirable properties that can be derived from the software specification or user expectations. We tested BWA, Bowtie and Bowtie2 using simulated data and one HapMap dataset. It is interesting to observe that multiple executions of the same aligner using slightly modified input FASTQ sequence file, such as after randomly re-ordering of the reads, may affect alignment results. Furthermore, we found that the list of variant calls can be affected unless strict quality control is applied during variant calling.ConclusionThorough testing of bioinformatics software is important in delivering clinical genomic medicine. This paper demonstrates a different framework to test a program that involves checking its properties, thus greatly expanding the number and repertoire of test cases we can apply in practice.

Mapping the non-standardized biases of ribosome profiling

“…Comparison of the tools is not trivial and differs depending on the data set, thus only few objective investigations have been performed so far (Giannoulatou et al, 2014). In the majority of ribosome profiling experiments (Ingolia et al, 2009(Ingolia et al, , 2011Guo et al, 2010;Gerashchenko et al, 2012;Li et al, 2012;Chew et al, 2013;Guttman et al, 2013;Aspden et al, 2014;Baudin-Baillieu et al, 2014;Bazzini et al, 2014;Subramaniam et al, 2014), Bowtie (Langmead et al, 2009) is used as a BWT-based mapping program.…”

Refinement Type Contracts for Verification of Scientific Investigative Software

“…This leaves the most scientifically important pieces of the code untested, and it is often difficult to determine equivalence of the software's output with the known result due to stochasticity or floating point arithmetic [49,30]. Meta-morphic testing has been proposed as an alternative means of testing software without an oracle, which involves testing specific properties which are required to hold [36,6,16,7]. This requires a deep knowledge of testing methodology and a code structure which facilitates such tests.…”

How to test bioinformatics software?

“…These programs have been tested and evaluated using many reference test data by the developers. In a recent study by Giannoulatou et al (2014) these sequence aligner programs were tested using MT. In their approach, they identified nine MRs. As an example, one of the MRs stated that random permutation of the input should not affect the alignment results.…”