Workflow Design Analysis for High Resolution Satellite Image Analysis

Paraskevakos, Ioannis; Turilli, Matteo; Gonçalves, Bento Collares; Lynch, Heather J.; Jha, Shantenu

doi:10.1109/escience.2019.00013

Cited by 5 publications

(5 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…GPUs are not utilized for almost an hour at the beginning of the execution and utilization decreases to 80% some time after half of the total execution was completed. Our analysis shows that RADICAL-Pilot's scheduler did not schedule GPU tasks at the start of the execution even if GPU resources were available [44]. .…”

Section: Experiments 2: Resource Utilizationmentioning

confidence: 90%

See 1 more Smart Citation

Comparing Workflow Application Designs for High Resolution Satellite Image Analysis

Al-Saadi¹,

Paraskevakos²,

Gonçalves³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Very High Resolution satellite and aerial imagery are used to monitor and conduct large scale surveys of ecological systems. Convolutional Neural Networks have successfully been employed to analyze such imagery to detect large animals and salient features. As the datasets increase in volume and number of images, utilizing High Performance Computing resources becomes necessary. In this paper, we investigate three task-parallel, data-driven workflow designs to support imagery analysis pipelines with heterogeneous tasks on HPC. We analyze the capabilities of each design when processing datasets from two use cases for a total of 4,672 satellite and aerial images, and 8.35 TB of data. We experimentally model the execution time of the tasks of the image processing pipelines.We perform experiments to characterize the resource utilization, total time to completion, and overheads of each design. Based on the model, overhead and utilization analysis, we show which design is best suited to scientific pipelines with similar characteristics.

show abstract

Section: Experiments 2: Resource Utilizationmentioning

confidence: 90%

“…Data sources, the software used for their analysis and replication guidelines can be found at [44,45].…”

Section: Discussionmentioning

confidence: 99%

Comparing Workflow Application Designs for High Resolution Satellite Image Analysis

Al-Saadi¹,

Paraskevakos²,

Gonçalves³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Developing scalable pipelines and workflows for HPC tasks involving large datasets has also been well-studied in the literature ( Farnes et al, 2018 ; Hendrix et al, 2016 ; Paraskevakos et al, 2019 ; Lyons et al., 2019 ). For example, the authors of ( Farnes et al, 2018 ) presented a technique for building scalable workflows for analyzing large volumes of satellite imagery data, while Lyons et al (2019) presented a system for analyzing workflows related to weather-sensing data.…”

Section: Performance Attributesmentioning

confidence: 99%

Data-driven scalable pipeline using national agent-based models for real-time pandemic response and decision support

Bhattacharya

Chen

Hoops

et al. 2022

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

This paper describes an integrated, data-driven operational pipeline based on national agent-based models to support federal and state-level pandemic planning and response. The pipeline consists of ( i) an automatic semantic-aware scheduling method that coordinates jobs across two separate high performance computing systems; ( ii) a data pipeline to collect, integrate and organize national and county-level disaggregated data for initialization and post-simulation analysis; ( iii) a digital twin of national social contact networks made up of 288 Million individuals and 12.6 Billion time-varying interactions covering the US states and DC; ( iv) an extension of a parallel agent-based simulation model to study epidemic dynamics and associated interventions. This pipeline can run 400 replicates of national runs in less than 33 h, and reduces the need for human intervention, resulting in faster turnaround times and higher reliability and accuracy of the results. Scientifically, the work has led to significant advances in real-time epidemic sciences.

show abstract

“…Developing scalable pipelines and workflows for HPC tasks involving large datasets has also been well-studied in literature [14], [21], [25], [29]. For example, the authors of [14] present a technique for building scalable workflows for analyzing large volumes of satellite imagery data, while [25] present a system for analyzing workflows related to weather-sensing data.…”

Section: Related Workmentioning

confidence: 99%

Scalable Epidemiological Workflows to Support COVID-19 Planning and Response

Machi

Bhattacharya

Hoops

et al. 2021

2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

View full text Add to dashboard Cite

The COVID-19 global outbreak represents the most significant epidemic event since the 1918 influenza pandemic. Simulations have played a crucial role in supporting COVID-19 planning and response efforts. Developing scalable workflows to provide policymakers quick responses to important questions pertaining to logistics, resource allocation, epidemic forecasts and intervention analysis remains a challenging computational problem. In this work, we present scalable high performance computing-enabled workflows for COVID-19 pandemic planning and response. The scalability of our methodology allows us to run fine-grained simulations daily, and to generate county-level forecasts and other counterfactual analysis for each of the 50 states (and DC), 3140 counties across the USA. Our workflows use a hybrid cloud/cluster system utilizing a combination of local and remote cluster computing facilities, and using over 20,000 CPU cores running for 6-9 hours every day to meet this objective. Our state (Virginia), state hospital network, our university, the DOD and the CDC use our models to guide their COVID-19 planning and response efforts. We began executing these pipelines March 25, 2020, and have delivered and briefed weekly updates to these stakeholders for over 30 weeks without interruption.

show abstract

Workflow Design Analysis for High Resolution Satellite Image Analysis

Cited by 5 publications

References 21 publications

Comparing Workflow Application Designs for High Resolution Satellite Image Analysis

Comparing Workflow Application Designs for High Resolution Satellite Image Analysis

Data-driven scalable pipeline using national agent-based models for real-time pandemic response and decision support

Scalable Epidemiological Workflows to Support COVID-19 Planning and Response

Contact Info

Product

Resources

About