Large-Scale Validation of Hypothesis Generation Systems via Candidate Ranking

Abstract

The first step of many research projects is to define and rank a short list of candidates for study. In the modern rapidity of scientific progress, some turn to automated hypothesis generation (HG) systems to aid this process. These systems can identify implicit or overlooked connections within a large scientific corpus, and while their importance grows alongside the pace of science, they lack thorough validation. Without any standard numerical evaluation method, many validate generalpurpose HG systems by rediscovering a handful of historical findings, and some wishing to be more thorough may run laboratory experiments based on automatic suggestions. These methods are expensive, time consuming, and cannot scale. Thus, we present a numerical evaluation framework for the purpose of validating HG systems that leverages thousands of validation hypotheses. This method evaluates a HG system by its ability to rank hypotheses by plausibility; a process reminiscent of human candidate selection. Because HG systems do not produce a ranking criteria, specifically those that produce topic models, we additionally present novel metrics to quantify the plausibility of hypotheses given topic model system output. Finally, we demonstrate that our proposed validation method aligns with real-world research goals by deploying our method within MOLIERE, our recent topic-driven HG system, in order to automatically generate a set of candidate genes related to HIVassociated neurodegenerative disease (HAND). By performing laboratory experiments based on this candidate set, we discover a new connection between HAND and Dead Box RNA Helicase 3 (DDX3). Reproducibility: code, validation data, and results can be found at sybrandt.com/2018/validation.

Are Abstracts Enough for Hypothesis Generation?

“…More recently, we proposed a number of metrics to evaluate a-c relevance in [16]. This work describes how a number of embedding-based relationships, further summarized in the following section, quantify the fruitfulness of an individual query.…”

“…Due to this conceptual limitation, many projects validate their system by simply rediscovering a handful of "gold-standard" connections [31], [32], [33], [34]. Some few projects show their utility beyond the gold-standard by incorporating expert analysis and experiments [18], [35], [16]. While these results are important to show real-world application areas for hypothesis generation, lab work is time consuming, expensive, and clearly does not scale for large validation sets.…”

“…We evaluate our results in terms of quality, using the hypothesis ranking techniques developed in [16], and discuss practical challenges in terms of memory consumption, runtime, and interpretability. We find that corpora with a higher median document length perform better than those with shorter documents and that this effect can be more substantial than simply adding more documents.…”

Inhibition of the Dead Box RNA Helicase 3 prevents HIV-1 Tat and cocaine-induced neurotoxicity by targeting microglia activation

“…In the current analysis we queried the list of all human genes downloaded from HUGO (30) paired with HIV associated neurocognitive disorder. The generated hypotheses were ranked based on a number of techniques described in (24). The hypotheses ranking represent the level of association each gene has with HAND.…”

“…In addition to our own ranking methods, the algorithmic pipeline relies on several recently developed scalable machine learning methods that have not been adopted by other knowledge discovery systems such as low-dimensional representation manifold learning and scalable probabilistic topic modeling. (23,24) . In the present article, we performed MOLIERE analysis for possible links of human proteins with HAND in the biomedical literature.…”