Transcription factor motif quality assessment requires systematic comparative analysis

Abstract

Transcription factor (TF) binding site prediction remains a challenge in gene regulatory research due to degeneracy and potential variability in binding sites in the genome. Dozens of algorithms designed to learn binding models (motifs) have generated many motifs available in research papers with a subset making it to databases like JASPAR, UniPROBE and Transfac. The presence of many versions of motifs from the various databases for a single TF and the lack of a standardized assessment technique makes it difficult for biologists to make an appropriate choice of binding model and for algorithm developers to benchmark, test and improve on their models. In this study, we review and evaluate the approaches in use, highlight differences and demonstrate the difficulty of defining a standardized motif assessment approach. We review scoring functions, motif length, test data and the type of performance metrics used in prior studies as some of the factors that influence the outcome of a motif assessment. We show that the scoring functions and statistics used in motif assessment influence ranking of motifs in a TF-specific manner. We also show that TF binding specificity can vary by source of genomic binding data. We also demonstrate that information content of a motif is not in isolation a measure of motif quality but is influenced by TF binding behaviour. We conclude that there is a need for an easy-to-use tool that presents all available evidence for a comparative analysis.

MARS: Motif Assessment and Ranking Suite for transcription factor binding motifs

“…We downloaded all ChIP-seq peaks uniformly processed by Analysis Working Group (AWG) from ENCODE [32], PBM from UniPROBE [5] database and PMW motifs from various databases and publications, prepared as previously described [18] and stored in a MySQL database ( Table 1). Alternative TF names (from GeneCards [29]) link various alternative TF names to a TF class ID derived from the TFClass classification [38] to find all motifs for a TF irrespective of naming inconsistency.…”

“…We have previously described the implementation of assessing by scoring and enrichment algorithms [18]. In summary, for each TF, the motifs in PWM format are used to score sequences partitioned into positive (test) and the negative (background) using one of the implemented scoring functions.…”

“…Finally, the ability to classify the two sets is evaluated by area under receiver operator characteristic curve (AUC) or the mean normalized conditional probability (MNCP) statistics ( Table 2). See our previous paper [18] for more details on the scoring functions and statistics used.…”

“…Nonetheless, there have been some attempts to develop tools and techniques to evaluate motif discovery algorithms, which can be categorized into assess-by-binding site prediction, motif comparison or by sequence scoring and classification [18]. We relate a selection of known approaches to the benchmark criteria we outline above.…”

“…The spread of motif models across DBs and in different PWM formats makes it difficult to create a benchmark that ranks multiple motifs for a given TF, and this problem is compounded by the growth in available data [18]. There is a lack of an easily accessible and independent motif evaluation platform that can allow users to rank PWM models for a given TF.…”