Bounds and Algorithms for Fast Exact Searches of Chemical Fingerprints in Linear and Sublinear Time.

Abstract

Chemical fingerprint are used to represent chemical molecules by recording the presence or absence, or by counting the number of occurrences, of particular features or substructures, such as labeled paths in the 2D graph of bonds, of the corresponding molecule. These fingerprint vectors are used to search large databases of small molecules, currently containing millions of entries, using various similarity measures, such as the Tanimoto or Tversky's measures and their variants. Here we derive simple bounds on these similarity measures, and show how these bounds can be used to considerably reduce the subset of molecules that need to be searched. We consider both the case of single-molecule and multiple-molecule queries, as well as queries based on fixed similarity thresholds or aimed at retrieving the top K hits. We study the speedup as a function of query size and distribution, fingerprint length, similarity threshold, and database size |D| and derive analytical formula that are in excellent agreement with empirical values. The theoretical considerations and experiments show that this approach can provide linear speedups of one or more orders of magnitude in the case of searches with fixed threshold, and achieve sublinear speedups in the range of O(|D| 0.6 ) for the top K hits in current large databases. This pruning approach yields subsecond search times across the 5M compounds in the ChemDB database, without any loss of accuracy.

Speeding Up Chemical Database Searches Using a Proximity Filter Based on the Logical Exclusive OR

“…We use the Tanimoto similarity because it is the most widely used measure. However, the methods to be described can be extended immediately to other similarity measures, 10,12 such as the Tversky similarity measure 13,14…”

“…This is because most other measures of similarity can also be expressed in terms of A∩B and A∪B (as well as obvious terms such as A, B, and N), 10 and the present approach relies on providing bounds for these individual components shared by most similarity measures found in the literature.…”

“…[3][4][5][6][7][8][9] The somewhat parallel expansions in data and representations create new opportunities and challenges and require the continued development of methods to store and search molecules in chemical databases in ways that can scale up with these expansions. One strategy 10 to speed up database searches based on molecular similarity is essentially a pruning strategy with two basic components. First, a small binary signature vector is extracted from each fingerprint and stored in the database together with the basic fingerprint vector.…”

“…More importantly, it should yield similarity upper bounds that can be computed rapidly, in order to sift rapidly through a large number of molecules, and that are tight, in order to prune a large number of compounds. Only the most simple signature was used in ref 10 to derive the upper bounds, consisting of a single number corresponding to the total number of 1-bits in the fingerprint vector. Even so, knowing the total number of 1-bits in the fingerprint yields simple bounds on standard similarity measures, such as the Tanimoto measure, which in turn can yield very significant amounts of pruning, considerably speeding up database searches.…”

“…Even so, knowing the total number of 1-bits in the fingerprint yields simple bounds on standard similarity measures, such as the Tanimoto measure, which in turn can yield very significant amounts of pruning, considerably speeding up database searches. 10 Here, we take this approach one step further by developing a new, more subtle, and complementary proximity filter, based on a signature derived using the exclusive-OR logical operator (XOR). The standard OR operator is used in several chemoinformatics systems, such as the Daylight, Avalon, and Unity systems, to lossly compress long fingerprints by application of the OR operator modulo n (e.g., 1024 bits) to derive short fingerprint representations.…”