Large-scale virtual screening on public cloud resources with Apache Spark

Abstract

Background Structure-based virtual screening is an in-silico method to screen a target receptor against a virtual molecular library. Applying docking-based screening to large molecular libraries can be computationally expensive, however it constitutes a trivially parallelizable task. Most of the available parallel implementations are based on message passing interface, relying on low failure rate hardware and fast network connection. Google’s MapReduce revolutionized large-scale analysis, enabling the processing of massive datasets on commodity hardware and cloud resources, providing transparent scalability and fault tolerance at the software level. Open source implementations of MapReduce include Apache Hadoop and the more recent Apache Spark.ResultsWe developed a method to run existing docking-based screening software on distributed cloud resources, utilizing the MapReduce approach. We benchmarked our method, which is implemented in Apache Spark, docking a publicly available target receptor against 2.2 M compounds. The performance experiments show a good parallel efficiency (87%) when running in a public cloud environment.ConclusionOur method enables parallel Structure-based virtual screening on public cloud resources or commodity computer clusters. The degree of scalability that we achieve allows for trying out our method on relatively small libraries first and then to scale to larger libraries. Our implementation is named Spark-VS and it is freely available as open source from GitHub (https://github.com/mcapuccini/spark-vs).Graphical abstract.

PySpark and RDKit: Moving towards Big Data in Cheminformatics

“…Scaling efficiency (Figure ) on the cluster presents comparable results to a virtual screening implementation of Spark, trending in an overall loss with an increase in vCPUs and a maximum scaling efficiency of ∼130 % at 10 vCPUs. The stabilization seems to originate in the communication overhead present in cluster operations.…”

“…Hence, the memory is not a limiting factor. Weak scaling efficiency was calculated according to prior methods, against an operation of the cluster with 1 vCPU. In the second experiment, we evaluated scaling effects on an increasing number of compounds i. e. 2, 5,10 million against 1 million with a fixed number of vCPUs set to 60.…”

“…Several implementations of big data computing in cheminformatics have been described in literature. One such is a multi‐server approach for descriptor calculation employing 337 computing nodes and using a dataset of 15000 compounds for descriptor calculation using in‐house developed, now freely available software, another is the virtual screening for 2.2 million compounds using Apache Spark at different levels of CPU utilization . However, there is still the need for efficient and freely (open code) accessible solutions for cheminformatics in the big data environment and distributed computing.…”

“…For text data (e. g. SMILES strings), the data would be split line by line, making SMILES convenient for distributing over virtual CPUs. Spark has proven to be useful in cheminformatics, particularly in large‐scale, parallel virtual screening . The afore mentioned implementation of Apache Spark showed that the SDF file format, even though multilined, can also conveniently be used for distributing into RDDs.…”

“…Spark has proven to be useful in cheminformatics, particularly in large‐scale, parallel virtual screening . The afore mentioned implementation of Apache Spark showed that the SDF file format, even though multilined, can also conveniently be used for distributing into RDDs. Moreover, the Python interface for Spark (PySpark), can be used together with RDKit on a Hadoop cluster with the aim of taking advantage of the processing framework capabilities.…”