Performance analysis and optimization of in-situ integration of simulation with data analysis

Fu, Yuankun; Li, Feng; Song, Fengguang

doi:10.1145/3208040.3208049

Cited by 8 publications

(5 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Researchers have been using "in-situ" to describe the situation when visualization/analysis programs can process simulation data without any data movements or if the visualization/analysis routines reside in the same processes with simulation [6,44]. There is also a broader definition of "in-situ" processing: processing data while it is generated [9,11,23,32]. More detailed terminology description and classification of in-situ has been introduced [10].…”

Section: In-situ Processing On Hpc Systemsmentioning

confidence: 99%

“…Data transport libraries such as ADIOS [32], Decaf [18], and Zipper [23] can be used to construct in-situ workflows that consist of separate parallel applications. Paraview Catalyst [2] and SENSEI [3] utilize the standard visualization toolkit (VTK) data representation, and allow simulation and visualization/analysis applications coupled through customized "adapters".…”

Section: Related Workmentioning

confidence: 99%

“…Such a data storage/movement/postanalysis pattern is often extremely expensive, and hence there is an inevitable trend to pursue in-situ data analysis. In a typical in-situ data analysis, the analysis applications continuously process/analyze the in-memory data structures while the simulation applications are running and generating data [6,16,18,23].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

X-composer

Wang

Yan

et al. 2021

Proceedings of the Platform for Advanced Scientific Computing Conference

Self Cite

View full text Add to dashboard Cite

As large-scale scientific simulations and big data analyses become more popular, it is increasingly more expensive to store huge amounts of raw simulation results to perform post-analysis. To minimize the expensive data I/O, "in-situ" analysis is a promising approach, where data analysis applications analyze the simulation generated data on the fly without storing it first. However, it is challenging to organize, transform, and transport data at scales between two semantically different ecosystems due to the distinct software and hardware difference. To tackle these challenges, we design and implement the X-Composer framework. X-Composer connects cross-ecosystem applications to form an "in-situ" scientific workflow, and provides a unified approach and recipe for supporting such hybrid in-situ workflows on distributed heterogeneous resources. X-Composer reorganizes simulation data as continuous data streams and feeds them seamlessly into the Cloud-based stream processing services to minimize I/O overheads. For evaluation, we use X-Composer to set up and execute a cross-ecosystem workflow, which consists of a parallel Computational Fluid Dynamics simulation running on HPC, and a distributed Dynamic Mode Decomposition analysis application running on Cloud. Our experimental results show that X-Composer can seamlessly couple HPC and Big Data jobs in their own native environments, achieve good scalability, and provide high-fidelity analytics for ongoing simulations in real-time.

show abstract

Section: In-situ Processing On Hpc Systemsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

X-composer

Wang

Yan

et al. 2021

Proceedings of the Platform for Advanced Scientific Computing Conference

Self Cite

View full text Add to dashboard Cite

show abstract

“…1b, component applications in an in-situ workflow run concurrently, exchanging data via network or shared memory. Workflow performance is determined by the complex interplay of the applications, which may involve factors such as load imbalances, contending network bandwidth, synchronizations, and locks [13]. High performance requires that the component applications execute in a balanced and coordinated way.…”

Section: Auto-tuning For In-situ Workflowsmentioning

confidence: 99%

“…The solution described in this section mainly targets looselycoupled in-situ workflows [13,25], in which components are coupled via high-level libraries (e.g., ADIOS [18], Flexpath [7], DataSpaces [9], FlexIO [33], GLEAN [28], DIMES [32], Decaf [10], or Zipper [13]). The relative ease of development and deployment of loosely-coupled in-situ workflows relative to tightly-coupled in-situ workflows makes the former much more prevalent.…”

Section: Generating the Low-fidelity Modelmentioning

confidence: 99%

In-situ Workflow Auto-tuning via Combining Performance Models of Component Applications

Shu¹,

Guo²,

Wozniak³

et al. 2020

Preprint

View full text Add to dashboard Cite

In-situ parallel workflows couple multiple component applications, such as simulation and analysis, via streaming data transfer in order to avoid data exchange via shared file systems. Such workflows are challenging to configure for optimal performance due to the large space of possible configurations. Expert experience is rarely sufficient to identify optimal configurations, and existing empirical auto-tuning approaches are inefficient due to the high cost of obtaining training data for machine learning models. It is also infeasible to optimize individual components independently, due to component interactions. We propose here a new autotuning method, Component-based Ensemble Active Learning (CEAL), that combines machine learning techniques with knowledge of in-situ workflow structure to enable automated workflow configuration with a limited number of performance measurements. Experiments with real applications demonstrate that CEAL can identify significantly better configurations than other approaches given compute time budgets. For example, with 50 training samples, it reduces execution time and computer time for a realistic workflow by 17.6% and 40.8% relative to random sampling, and by 12.4% and 32.5% relative to a state-of-the-art algorithm, GEIST, respectively. CEAL is also cost-effective: The tuned workflow need be run only 864 times to pay off training sample collection costs, 40% less than the 1444 times required with pure active learning.

show abstract

INSTANT: A Runtime Framework to Orchestrate In-Situ Workflows

Li,

Song

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Performance analysis and optimization of in-situ integration of simulation with data analysis

Cited by 8 publications

References 27 publications

X-composer

X-composer

In-situ Workflow Auto-tuning via Combining Performance Models of Component Applications

INSTANT: A Runtime Framework to Orchestrate In-Situ Workflows

Contact Info

Product

Resources

About