Selective and Recurring Re-computation of Big Data Analytics Tasks: Insights from a Genomics Case Study

Abstract

In Data Science, knowledge generated by a resource-intensive analytics process is a valuable asset. Such value, however, tends to decay over time as a consequence of the evolution of any of the elements the process depends on: external data sources, libraries, and system dependencies. It is therefore important to be able to (i) detect changes that may partially or completely invalidate prior outcomes, (ii) determine the impact that those changes will have on those prior outcomes, ideally without having to perform expensive re-computations, and (iii) optimise the process re-execution needed to selectively refresh affected outcomes. This paper presents an extensive experimental study on how the selective re-computation problem manifests itself in a relevant analytics task for Genomics, namely variant calling and clinical interpretation, and how the problem can be addressed using a combination of approaches. Starting from this experience, we then offer a blueprint for a generic re-computation meta-process that makes use of process history metadata to make informed decisions about selective recomputations in reaction to a variety of changes in the data.

Countering Terrorism on Social Media Using Big Data

“…Dealing with all this data is challenging as traditional methods of analysis such as guessing, constructing hypothesis and testing with data based experiments do not perform well [22] because of the sheer volume and variety of data, hence new methods uncovering the insights in data must be devised. Research in Big Data is considering new techniques [23][24][25][26][27] with the core challenge of how to process data to extract useful information. Computational resources seem to be second runners-up in this race with Big Data leading the way.…”

“…Computational resources seem to be second runners-up in this race with Big Data leading the way. Classic learning and intelligence methods also fail to perform well on large data set, and new "Deep Learning" techniques [23,24] are being devised to take advantage of the available information.…”