Systematic evaluation of connectivity map for disease indications

Abstract

BackgroundConnectivity map data and associated methodologies have become a valuable tool in understanding drug mechanism of action (MOA) and discovering new indications for drugs. One of the key ideas of connectivity map (CMAP) is to measure the connectivity between disease gene expression signatures and compound-induced gene expression profiles. Despite multiple impressive anecdotal validations, only a few systematic evaluations have assessed the accuracy of this aspect of CMAP, and most of these utilize drug-to-drug matching to transfer indications across the two drugs.MethodsTo assess CMAP methodologies in a more direct setting, namely the power of classifying known drug-disease relationships, we evaluated three CMAP-based methods on their prediction performance against a curated dataset of 890 true drug-indication pairs. The disease signatures were generated using Gene Logic BioExpress™ system and the compound profiles were derived from the Connectivity Map database (CMAP, build 02, http://www.broadinstitute.org/CMAP/).ResultsThe similarity scoring algorithm called eXtreme Sum (XSum) performs better than the standard Kolmogorov-Smirnov (KS) statistic in terms of the area under curve and can achieve a four-fold enrichment at 0.01 false positive rate level, with AUC = 2.2E-4, P value = 0.0035.ConclusionConnectivity map can significantly enrich true positive drug-indication pairs given an effective matching algorithm.Electronic supplementary materialThe online version of this article (doi:10.1186/s13073-014-0095-1) contains supplementary material, which is available to authorized users.

A review of connectivity map and computational approaches in pharmacogenomics

“…Transcriptional expression profiles are widely used to find drug-disease or drug-drug relationships that could lead to new methods in drug discovery [28]. However, a remaining challenge is to evaluate methods based on such data sets.…”

“…Despite the success of various CMap approaches, there are few ways to quantitatively evaluate the performance of the connectivity score for the association between drugs and diseases by computational means. There are two ways to computationally evaluate CMap: first, evaluate drug-drug relations [18,42] and second, evaluate disease-drug relations [28].…”

“…However, in evaluating disease-drug relations, a disease signature is used to query CMap to retrieve known drugs notably in [28].…”

“…Second, the benchmark sets used to represent the drugdisease association might not be comprehensive enough to capture all drug-disease linkages. Finally, the drug cellular profiles are limited to only treating fewer cell lines, which explains why some of the neoplastic disease signatures perform better than nonneoplastic disease signatures [28].…”

“…The rest of the expression sets were profiled in a B-cell lymphoma cell lines, which are chronic lymphocytic leukemia patient-derived cell lines(GSE14973), K422 non-Hodgkin's lymphoma cell lines (GSE7292), lytic-permissive lymphoblastoid cell lines (GSE31447), diffuse large B-cell lymphoma patient-derived cell lines (GSE40003) and mantle cell lymphoma cell lines (GSE34602 [24][25][26][27]. Notably, Cheng et al [28] presented a systematic approach to quantitatively assess the performance of such methods. Hence, this study can be seen as a benchmark approach to assess any new methodology in the future.…”