Towards Continuous Scientific Data Analysis and Hypothesis Evolution

Abstract

Scientific data is continuously generated throughout the world. However, analyses of these data are typically performed exactly once and on a small fragment of recently generated data. Ideally, data analysis would be a continuous process that uses all the data available at the time, and would be automatically re-run and updated when new data appears. We present a framework for automated discovery from data repositories that tests user-provided hypotheses using expert-grade data analysis strategies, and reassesses hypotheses when more data becomes available. Novel contributions of this approach include a framework to trigger new analyses appropriate for the available data through lines of inquiry that support progressive hypothesis evolution, and a representation of hypothesis revisions with provenance records that can be used to inspect the results. We implemented our approach in the DISK framework, and evaluated it using two scenarios from cancer multi-omics: 1) data for new patients becomes available over time, 2) new types of data for the same patients are released. We show that in all scenarios DISK updates the confidence on the original hypotheses as it automatically analyzes new data.

Beyond Reproduction, Experiments want to be Understood

“…But it may also be useful to apply the same analysis on different (or updated) data. For instance, data analysis workflows can be exploited for continuous reevaluation of hypotheses by updating data analyses when new data is available [10].…”

Will AI write scientific papers in the future?

“…We are developing a framework, called DISK, to make hypothesis testing and data analysis more systematic (Gil et al 2017a). We look at the discovery cycle, starting with formulating new hypotheses, determining what type of data and method can be used to test it (we call this a line of inquiry), retrieving the data from a shared repository, analyzing the data, and then revising the hypothesis.…”

On Reproducible AI: Towards Reproducible Research, Open Science, and Digital Scholarship in AI Publications

“…For example, a citation for a bundle containing workflows and execution details for Gil et al (2017) is:…”

“…A citation for a dataset consists of a descriptive name (or title) for the dataset, its creators, the name of the repository where it can be accessed, and the permanent URL. For example, a citation for a dataset in Gil et al (2017) is: Adusumilli, Ravali. (2016).…”

“…A citation for a source code version consists of a descriptive name (or title) for the source code, its creators, the name of the repository where it can be accessed, the version, and the permanent URL. For example, a citation for GitHub code in (Gil et al 2017) is: By uploading the source code into the GitHub code repository, we obtained a persistent identifier for the version used in the publication as well as the citation. Specifying the authors, the name, and the license takes negligible effort.…”