Combining in-situ and in-transit processing to enable extreme-scale scientific analysis

Bennett, Janine Camille; Abbasi, Hasan; Bremer, Peer-Timo; Grout, Ray; Gyulassy, Attila; Jin, Tong; Klasky, Scott; Kolla, Hemanth; Parashar, Manish; Pascucci, Valerio; Pébaÿ, Philippe Pierre; Thompson, David C.; Yu, Hongfeng; Zhang, Fan; Chen, Jacqueline H.

doi:10.1109/sc.2012.31

Cited by 138 publications

(77 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Furthermore, work has been done in the broad research area of in situ visualization (e.g., the state-of-the-art analysis by Rivi et al [4]). The applications range from visualizing combustion problems (Yu et al [5]) to general procedures and software solutions (Fabian et al [6]) and extreme-scale scientific analysis (Bennett et al [7]). From reviewing these platforms and concepts, the authors decided to develop a new data structure, allowing one to solve some of the mentioned shortcomings and to provide a flexible environment for further model implementations.…”

Section: Introductionmentioning

confidence: 99%

Engineering-Based Thermal CFD Simulations on Massive Parallel Systems

et al. 2015

View full text Add to dashboard Cite

The development of parallel Computational Fluid Dynamics (CFD) codes is a challenging task that entails efficient parallelization concepts and strategies in order to achieve good scalability values when running those codes on modern supercomputers with several thousands to millions of cores. In this paper, we present a hierarchical data structure for massive parallel computations that supports the coupling of a Navier-Stokes-based fluid flow code with the Boussinesq approximation in order to address complex thermal scenarios for energy-related assessments. The newly designed data structure is specifically designed with the idea of interactive data exploration and visualization during runtime of the simulation code; a major shortcoming of traditional high-performance computing (HPC) simulation codes. We further show and discuss speed-up values obtained on one of Germany's top-ranked supercomputers with up to 140,000 processes and present simulation results for different engineering-based thermal problems.

show abstract

Section: Introductionmentioning

confidence: 99%

Engineering-Based Thermal CFD Simulations on Massive Parallel Systems

et al. 2015

View full text Add to dashboard Cite

show abstract

“…NVRAM will likely have a considerable impact on the storage landscape, but most of the known candidates are not ready for mass production or remain too expensive to replace other technologies [4]. Many data transformations such as compression or encryption could be performed out-of-core, either on GPUs or in-transit [2,42].…”

Section: Technologiesmentioning

confidence: 99%

Survey of Storage Systems for High-Performance Computing

Lüttgau

Kuhn

Duwe

et al. 2018

JSFI

View full text Add to dashboard Cite

In current supercomputers, storage is typically provided by parallel distributed file systems for hot data and tape archives for cold data. These file systems are often compatible with local file systems due to their use of the POSIX interface and semantics, which eases development and debugging because applications can easily run both on workstations and supercomputers. There is a wide variety of file systems to choose from, each tuned for different use cases and implementing different optimizations. However, the overall application performance is often held back by I/O bottlenecks due to insufficient performance of file systems or I/O libraries for highly parallel workloads. Performance problems are dealt with using novel storage hardware technologies as well as alternative I/O semantics and interfaces. These approaches have to be integrated into the storage stack seamlessly to make them convenient to use. Upcoming storage systems abandon the traditional POSIX interface and semantics in favor of alternative concepts such as object and key-value storage; moreover, they heavily rely on technologies such as NVM and burst buffers to improve performance. Additional tiers of storage hardware will increase the importance of hierarchical storage management. Many of these changes will be disruptive and require application developers to rethink their approaches to data management and I/O. A thorough understanding of today's storage infrastructures, including their strengths and weaknesses, is crucially important for designing and implementing scalable storage systems suitable for demands of exascale computing.

show abstract

“…Here, we analyze a development version of a massively parallel algorithm [27] to compute merge trees, a topological structure that has recently been used to analyze some of the largest combustion simulations [8,5,4]. The algorithm relies on a global gather-scatter approach similar to a V-cycle in a traditional multi-grid solver with the corresponding problem of low parallel efficiency as most processes are kept idle some of the time.…”

Section: Massively Parallel Merge Treesmentioning

confidence: 99%

Combing the Communication Hairball: Visualizing Parallel Execution Traces using Logical Time

Isaacs

Bremer

Jusufi

et al. 2014

IEEE Trans. Visual. Comput. Graphics

Self Cite

View full text Add to dashboard Cite

Fig. 1: Logical timeline and clustered logical timeline views from Ravel, a tool for visualizing parallel execution traces. Events are represented by boxes, colored by their wall-clock delay. The use of logical time reveals communication patterns and leverages developers' understanding of their program's structure. We use the logical time structure to cluster on any metric, which allow us to represent large-scale traces using explorable clusters while still depicting messages with full timelines for a subset of processes.Abstract-With the continuous rise in complexity of modern supercomputers, optimizing the performance of large-scale parallel programs is becoming increasingly challenging. Simultaneously, the growth in scale magnifies the impact of even minor inefficiencies -potentially millions of compute hours and megawatts in power consumption can be wasted on avoidable mistakes or sub-optimal algorithms. This makes performance analysis and optimization critical elements in the software development process. One of the most common forms of performance analysis is to study execution traces, which record a history of per-process events and interprocess messages in a parallel application. Trace visualizations allow users to browse this event history and search for insights into the observed performance behavior. However, current visualizations are difficult to understand even for small process counts and do not scale gracefully beyond a few hundred processes. Organizing events in time leads to a virtually unintelligible conglomerate of interleaved events and moderately high process counts overtax even the largest display. As an alternative, we present a new trace visualization approach based on transforming the event history into logical time inferred directly from happened-before relationships. This emphasizes the code's structural behavior, which is much more familiar to the application developer. The original timing data, or other information, is then encoded through color, leading to a more intuitive visualization. Furthermore, we use the discrete nature of logical timelines to cluster processes according to their local behavior leading to a scalable visualization of even long traces on large process counts. We demonstrate our system using two case studies on large-scale parallel codes.

show abstract

Combining in-situ and in-transit processing to enable extreme-scale scientific analysis

Cited by 138 publications

References 35 publications

Engineering-Based Thermal CFD Simulations on Massive Parallel Systems

Engineering-Based Thermal CFD Simulations on Massive Parallel Systems

Survey of Storage Systems for High-Performance Computing

Combing the Communication Hairball: Visualizing Parallel Execution Traces using Logical Time

Contact Info

Product

Resources

About