Compressive genomics

Loh, Po−Ru; Baym, Michael; Berger, Bonnie

doi:10.1038/nbt.2241

Cited by 109 publications

(108 citation statements)

References 24 publications

Supporting

Mentioning

107

Contrasting

Unclassified

Order By: Relevance

“…The primary ways the literature addresses this problem—locality sensitive hashing (Indyk & Motwani, 1998), vector approximation (Ferhatosmanoglu et al, 2000), and space partitioning (Weber et al, 1998)—involve the construction of data structures that support more efficient search operations. However, we note that as biological data increases, not only does the redundancy present in the data also increase (Loh et al, 2012), but internal structure (such as the fact that not all conceivable configurations, e.g. all possible protein sequences, actually exist) also becomes apparent.…”

Section: Introductionmentioning

confidence: 99%

Entropy-Scaling Search of Massive Biological Data

Daniels

Danko

et al. 2015

Cell Systems

Self Cite

View full text Add to dashboard Cite

Summary Many data sets exhibit well-defined structure that can be exploited to design faster search tools, but it is not always clear when such acceleration is possible. Here we introduce a framework for similarity search based on characterizing a data set’s entropy and fractal dimension. We prove that searching scales in time with metric entropy (number of covering hyperspheres), if the fractal dimension of the data set is low, and scales in space with the sum of metric entropy and information-theoretic entropy (randomness of the data). Using these ideas, we present accelerated versions of standard tools, with no loss in specificity and little loss in sensitivity, for use in three domains—high-throughput drug screening (Ammolite, 150x speedup), metagenomics (MICA, 3.5x speedup of DIAMOND (3700x BLASTX)), and protein structure search (esFragBag, 10x speedup of FragBag). Our framework can be used to achieve ‘compressive omics,’ and the general theory can be readily applied to data science problems outside of biology. Source code: http://gems.csail.mit.edu

show abstract

Section: Introductionmentioning

confidence: 99%

Entropy-Scaling Search of Massive Biological Data

Daniels

Danko

et al. 2015

Cell Systems

Self Cite

View full text Add to dashboard Cite

show abstract

“…1. Скорость роста вычислительных мощностей компьютеров (зеленый) и новых последовательностей ДНК (синий) [11].…”

Section: медицина и биологияunclassified

The Problem of Processing and Storage of Large Amounts of Scientific Data and Approaches to Its Solution

Анатольевич¹,

Вячеславович²

2013

Math.Biol.Bioinf.

View full text Add to dashboard Cite

Аннотация. В статье рассматривается актуальная проблема значительного роста объёмов данных, получаемых, хранимых и обрабатываемых в ходе научной деятельности, в первую очередь в таких областях как биоинформатика и астрофизика. Рассматриваются современные программные методы и компьютерные технологии, используемые для работы со сверхбольшими объёмами данных. Проводится анализ состояния дел в институтах Пущинского научного центра РАН -Институте математических проблем биологии РАН и Пущинской радиоастрономической обсерватории Астрокосмического центра ФИАН им. Лебедева.

show abstract

“…The total amount of available genomic data is increasing approximately 10-fold every year, a rate much faster than described by Moore's Law for computational processing power (Loh et al, 2012). This holds true for resequencing studies.…”

Section: Introductionmentioning

confidence: 98%

ReSeqTools: an integrated toolkit for large-scale next-generation sequencing based resequencing analysis

Zhao²,

Liu³

et al. 2013

Genet. Mol. Res.

View full text Add to dashboard Cite

ABSTRACT. Large-scale next-generation sequencing (NGS)-based resequencing detects sequence variations, constructs evolutionary histories, and identifies phenotype-related genotypes. However, NGSbased resequencing studies generate extraordinarily large amounts of data, making computations difficult. Effective use and analysis of these data for NGS-based resequencing studies remains a difficult task for individual researchers. Here, we introduce ReSeqTools, a full-featured toolkit for NGS (Illumina sequencing)-based resequencing analysis, which processes raw data, interprets mapping results, and identifies and annotates sequence variations. ReSeqTools provides abundant scalable functions for routine resequencing analysis in different modules to facilitate customization of the analysis pipeline. ReSeqTools is designed to use compressed data files as input or output to save storage space and facilitates faster and more computationally efficient large-scale resequencing studies in a user-friendly manner. It offers abundant practical functions and generates useful statistics during the analysis pipeline, which significantly simplifies resequencing analysis. Its integrated algorithms and abundant sub-functions provide a solid foundation for special demands in resequencing projects. Users can combine these functions to construct their own pipelines for other purposes.

show abstract

Compressive genomics

Cited by 109 publications

References 24 publications

Entropy-Scaling Search of Massive Biological Data

Entropy-Scaling Search of Massive Biological Data

The Problem of Processing and Storage of Large Amounts of Scientific Data and Approaches to Its Solution

ReSeqTools: an integrated toolkit for large-scale next-generation sequencing based resequencing analysis

Contact Info

Product

Resources

About