Entropy-Scaling Search of Massive Biological Data

Abstract

Summary Many data sets exhibit well-defined structure that can be exploited to design faster search tools, but it is not always clear when such acceleration is possible. Here we introduce a framework for similarity search based on characterizing a data set’s entropy and fractal dimension. We prove that searching scales in time with metric entropy (number of covering hyperspheres), if the fractal dimension of the data set is low, and scales in space with the sum of metric entropy and information-theoretic entropy (randomness of the data). Using these ideas, we present accelerated versions of standard tools, with no loss in specificity and little loss in sensitivity, for use in three domains—high-throughput drug screening (Ammolite, 150x speedup), metagenomics (MICA, 3.5x speedup of DIAMOND (3700x BLASTX)), and protein structure search (esFragBag, 10x speedup of FragBag). Our framework can be used to achieve ‘compressive omics,’ and the general theory can be readily applied to data science problems outside of biology. Source code: http://gems.csail.mit.edu

Online Databases for Taxonomy and Identification of Pathogenic Fungi and Proposal for a Cloud-Based Dynamic Data Network Platform

“…While a single sequence alignment against a reference database of a few million sequences may take a few milliseconds to 2 to 3 s on a given computer, the comparison of millions of sequences against millions of references will cause serious scalability problems and challenges. Faster heuristic comparative methods (56,57) will have to be developed and implemented in a cloudor grid-based environment.…”

Functional association prediction by community profiling

“…To build the community profile of a given gene/protein sequence (herein referred to as the query sequence ), RAPSearch2 [29] (other fast similarity search tools including Diamond [30] and MICA [31] can also be utilized) is applied to search the sequence against a set of metagenomes. As described in details below, the set of metagenomes (herein referred to as the reference metagenomes ) were downloaded from metagenomic data repositories.…”

Computational biology in the 21st century

“…38 The first critical observation is that much biological data is highly redundant; if a computation is performed on one human genome, and a researcher wishes to perform the same computation on another human genome, most of the work has already been done. 22 When dealing with redundant data, clustering comes to mind.…”

“…38 That is, for a given cluster radius r c and a database D , the number k of clusters needed to cover D is bounded by N r c ( D ), the metric entropy, which is relatively small compared to | D |, the number of entries in the database (Figure 3). In contrast, if the points were uniformly distributed about the Cartesian space, N r c ( D ) would be larger.…”

“…38 That is, within some range of radii r 1 and r 2 about an arbitrary point in the database D , the fractal dimension d is $d=\frac{false(\text{log}false(n_2/n_{1}false)}{false(\text{log }false(r_2/r_{1}false)}$, where n 1 and n 2 are the number of points within r 1 and r 2 respectively (Figure 3). …”