Managing Large SNP Datasets with SNPpy

Mitha, Faheem

doi:10.1007/978-1-62703-447-0_4

Cited by 2 publications

(3 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, there are a few other software packages available that could possibly also be used for the intended purpose of TheSNPpit. SNPpy was already published in 2011 with an update in 2013 [5, 15], while dbVOR is much more recent from 2015 [6]. The design of the packages is quite different from TheSNPpit.…”

Section: Discussionmentioning

confidence: 99%

TheSNPpit—A High Performance Database System for Managing Large Scale SNP Data

Groeneveld

Lichtenberg

2016

PLoS ONE

View full text Add to dashboard Cite

The fast development of high throughput genotyping has opened up new possibilities in genetics while at the same time producing considerable data handling issues. TheSNPpit is a database system for managing large amounts of multi panel SNP genotype data from any genotyping platform. With an increasing rate of genotyping in areas like animal and plant breeding as well as human genetics, already now hundreds of thousand of individuals need to be managed. While the common database design with one row per SNP can manage hundreds of samples this approach becomes progressively slower as the size of the data sets increase until it finally fails completely once tens or even hundreds of thousands of individuals need to be managed. TheSNPpit has implemented three ideas to also accomodate such large scale experiments: highly compressed vector storage in a relational database, set based data manipulation, and a very fast export written in C with Perl as the base for the framework and PostgreSQL as the database backend. Its novel subset system allows the creation of named subsets based on the filtering of SNP (based on major allele frequency, no-calls, and chromosomes) and manually applied sample and SNP lists at negligible storage costs, thus avoiding the issue of proliferating file copies. The named subsets are exported for down stream analysis. PLINK ped and map files are processed as in- and outputs. TheSNPpit allows management of different panel sizes in the same population of individuals when higher density panels replace previous lower density versions as it occurs in animal and plant breeding programs. A completely generalized procedure allows storage of phenotypes. TheSNPpit only occupies 2 bits for storing a single SNP implying a capacity of 4 mio SNPs per 1MB of disk storage. To investigate performance scaling, a database with more than 18.5 mio samples has been created with 3.4 trillion SNPs from 12 panels ranging from 1000 through 20 mio SNPs resulting in a database of 850GB. The import and export performance scales linearly with the number of SNPs and is largely independent of panel and database size. Import speed is around 6 mio SNPs/sec, export between 60 and 120 mio SNPs/sec. Being command line based, imports and exports can easily be integrated into pipelines. TheSNPpit is available under the Open Source GNU General Public License (GPL) Version 2.

show abstract

Section: Discussionmentioning

confidence: 99%

TheSNPpit—A High Performance Database System for Managing Large Scale SNP Data

Groeneveld

Lichtenberg

2016

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…We have compared dbVOR to SNPpy, a recently developed database system [ 6 , 7 ]. Both SNPpy and dbVOR are database systems that store and retrieve genotype information from experiments.…”

Section: Discussionmentioning

confidence: 99%

“…While several database systems have been developed for managing genetic data[ 1 - 7 ], when we tried some of these, we found that some relied on commercial database systems that were so complicated that they required a database administrator to routinely maintain and apply regular security updates. Others did not scale well as the numbers of markers genotyped per experiment rapidly increased.…”

Section: Introductionmentioning

confidence: 99%

dbVOR: a database system for importing pedigree, phenotype and genotype data and exporting selected subsets

et al. 2015

View full text Add to dashboard Cite

BackgroundWhen studying the genetics of a human trait, we typically have to manage both genome-wide and targeted genotype data. There can be overlap of both people and markers from different genotyping experiments; the overlap can introduce several kinds of problems. Most times the overlapping genotypes are the same, but sometimes they are different. Occasionally, the lab will return genotypes using a different allele labeling scheme (for example 1/2 vs A/C). Sometimes, the genotype for a person/marker index is unreliable or missing. Further, over time some markers are merged and bad samples are re-run under a different sample name. We need a consistent picture of the subset of data we have chosen to work with even though there might possibly be conflicting measurements from multiple data sources.ResultsWe have developed the dbVOR database, which is designed to hold data efficiently for both genome-wide and targeted experiments. The data are indexed for fast retrieval by person and marker. In addition, we store pedigree and phenotype data for our subjects. The dbVOR database allows us to select subsets of the data by several different criteria and to merge their results into a coherent and consistent whole. Data may be filtered by: family, person, trait value, markers, chromosomes, and chromosome ranges. The results can be presented in columnar, Mega2, or PLINK format.ConclusionsdbVOR serves our needs well. It is freely available from https://watson.hgen.pitt.edu/register. Documentation for dbVOR can be found at https://watson.hgen.pitt.edu/register/docs/dbvor.html.Electronic supplementary materialThe online version of this article (doi:10.1186/s12859-015-0505-4) contains supplementary material, which is available to authorized users.

show abstract

Managing Large SNP Datasets with SNPpy

Cited by 2 publications

References 3 publications

TheSNPpit—A High Performance Database System for Managing Large Scale SNP Data

TheSNPpit—A High Performance Database System for Managing Large Scale SNP Data

dbVOR: a database system for importing pedigree, phenotype and genotype data and exporting selected subsets

Contact Info

Product

Resources

About