Adaptive Optimizations for Stream-based Workflows

Abstract

This work presents three new adaptive optimization techniques to maximize the performance of dispel4py workflows. dispel4py is a parallel Python-based stream-orientated dataflow framework that acts as a bridge to existing parallel programming frameworks like MPI or Python multiprocessing. When a user runs a dispel4py workflow, the original framework performs a fixed workload distribution among the processes available for the run. This allocation does not take into account workflows' features, which can cause scalability issues, specially for data-intensive scientific workflows. Therefore, our aim is to improve the performance of dispel4py workflows by testing different workload strategies that automatically adapt to workflows. For achieving this objective, we have implemented three new techniques, called Naive Assignment, Staging and Dynamic Scheduling. The evaluations show that our proposed techniques have significantly improved the performance of the original dispel4py framework.

Scalable adaptive optimizations for stream-based workflows in multi-HPC-clusters and cloud infrastructures

“…As we outlined in the conclusions of our previous work [35], we have taken this opportunity to test our optimizations in the context of DARE, in which several domain scientists have developed their applications using dispel4py. In this work, we run part of the Rapid Ground Motion Assessment application, developed by the Seismology community, with and without static optimizations using a Cloud environment.…”