Simplified parallel domain traversal

Abstract

Many data-intensive scientific analysis techniques require global domain traversal, which over the years has been a bottleneck for efficient parallelization across distributedmemory architectures. Inspired by MapReduce and other simplified parallel programming approaches, we have designed DStep, a flexible system that greatly simplifies efficient parallelization of domain traversal techniques at scale. In order to deliver both simplicity to users as well as scalability on HPC platforms, we introduce a novel two-tiered communication architecture for managing and exploiting asynchronous communication loads. We also integrate our design with advanced parallel I/O techniques that operate directly on native simulation output. We demonstrate DStep by performing teleconnection analysis across ensemble runs of terascale atmospheric CO2 and climate data, and we show scalability results on up to 65,536 IBM BlueGene/P cores.

Comparative visualization of vector field ensembles based on longest common subsequence

“…Initially, the pathlines are traced by numerical integration, namely, the Runge-Kutta method. Our algorithm is parallelized with a MapReduce-like framework [12] in order to efficiently handle large-size ensemble data. Then the parallel LCSS sequence encoding and the parallel LCSS distance computation are integrated in the framework.…”

“…We integrate the LCSS computation into a modified version of the DStep framework [12] to boost the efficiency of pathline tracing. It can achieve high scalability by its amphibious scheme on dataparallel and task-parallel, making the most intensive computation (i.e., pathline tracing) more efficient and effective.…”

Scalable parallel building blocks for custom data analysis

“…It features a simplified programming interface and a fixed-function analysis pipeline consisting of a local computation (map) followed by a global reduction (reduce). The DStep [18] model expands on MapReduce by adding a domain traversal stage prior to reduction. In contrast to fixed pipelines such as MapReduce and DStep, our approach requires more programming effort but allows arbitrary combinations of computation, global reduction, and local neighborhood communication.…”

“…Within each round (the outermost loop encompassing lines [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19], two inner loops are performed. The first is over the blocks that have not already been merged into a larger block, or active blocks (lines 2-13).…”

“…One block is designated to be the root and takes ownership of the merged result of that group; other blocks in the group are sent to the root. The second loop (lines [15][16][17][18] iterates over the groups (identified by their root block) and computes the merged result for each group.…”