Similarity Search for an Extreme Application: Experience and Implementation

Learned Indexing in Proteins: Substituting Complex Distance Calculations with Embedding and Clustering Techniques

“…We have chosen to test our approach on 3D protein structures for several reasons. First, while protein structure data is very widely used, and the study of this data is vital for almost every area of biochemical research, the issue of efficient search and comparison of protein structures is still unresolved to some extent, with many databases still relying on time-consuming brute-force linear search [14]. This data is also publicly available in a single database, called the Protein Data Bank (PDB), which is used by the majority of protein researchers and widely agreed upon as the standard.…”

“…We evaluated our approach using range queries, with 512 randomly chosen protein chains from the dataset used as query objects. In order to compare our results against the ground truth, we needed to know the Q distances (based on Q score ) between the 512 protein chains and all the other chains in the database -these distances were kindly provided by the researchers behind [14], where the same 512 objects were used as the pivots for their search engine. The objects were chosen uniformly randomly with respect to protein chain length, which ensures that even very long proteins are represented among our queries (despite constituting a relatively small portion of the dataset).…”

“…To examine this effect, we have chosen three representative query ranges of 0.1, 0.3 and 0.5 -in a real use case, the range would be chosen by a domain expert based on the particular use case. As a rule of thumb, a range of 0.1 represents a high degree of similarity, while a range of 0.5 represents low (but still biologically significant) similarity; the biological relevance of answers drops sharply beyond this range [14].…”

“…Distance computations on long proteins pose a considerable challenge for similarity searching methods, typically requiring a disproportionate amount of computing time [14]. This does not apply to our embedding approach, which transforms all the proteins into fixed-length vectors using the same algorithm, regardless of their length.…”

“…Finally, to provide broader context for the pipeline's performance, we have evaluated it against a more conventional approach. While there is no analogous method for searching 3D protein structures using range queries, there is a similar, recently-published method for nearest neighbor retrieval in the same database of 3D protein structures [14]. This method uses a three-stage search engine which compares bit-strings in the Hamming space ("sketches") to approximate the distance of protein chains.…”