Transparent Incremental Updates for Genomics Data Analysis Pipelines

Abstract

Abstract.A large up-to-date compendium of integrated genomic data is often required for biological data analysis. The compendium can be tens of terabytes in size, and must often be frequently updated with new experimental or metadata. Manual compendium update is cumbersome, requires a lot of unnecessary computation, and it may result in errors or inconsistencies in the compendium. We propose a transparent file based approach for adding incremental update capabilities to unmodified genomics data analysis tools and pipeline workflow managers. This approach is implemented in the GeStore system. We evaluate GeStore using a real world genomics compendium. Our results show that it is easy to add incremental updates to genomics data processing pipelines, and that incremental updates can reduce the computation time such that it becomes practical to maintain large-scale up-to-date genomics compendia on small clusters.

Data-Intensive Computing Infrastructure Systems for Unmodified Biological Data Analysis Pipelines

“…We achieved up to 82% reduction in analysis time for compendium updates when using GeStore with an unmodified biological data analysis pipeline ( [34] has additional experimental results). We found HBase to be well suited for the data management requirements of GeStore.…”

“…GeStore [34] is a framework for adding transparent incremental updates to data processing pipelines. We use GeStore to incrementally update large-scale compendia such as the IMP compendia described in the previous section.…”

Integrating Data-Intensive Computing Systems with Biological Data Analysis Frameworks

“…GeStore [38] is a system for adding transparent incremental updates to biological data processing pipelines. We use GeStore to periodically update large-scale compendia, such as the IMP compendia described in the previous section.…”

“…We built GeStore since the processing time for a full compendium update can be several days even on a large computer cluster, making it impractical to frequently update large-scale compendia. We have achieved up to 82% reduction in analysis time for dataset updates when using GeStore with an unmodified biological data analysis pipeline [38]. GeStore also provides efficient meta-database management for large scale meta-databases.…”

“…The configuration can specify that a dataset should be sampled with a given sample size. Mario uses the GeStore transparent file-based approach for incremental processing [38], so it is not necessary to modify tool code. The user then starts the initial computation.…”