Science Pipelines for the Square Kilometre Array

Farnes, J. S.; Mort, Ben; Dulwich, Fred; Salvini, Stefano; Armour, Wes

doi:10.20944/preprints201810.0115.v2

Cited by 10 publications

(5 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Developing scalable pipelines and workflows for HPC tasks involving large datasets has also been well-studied in the literature ( Farnes et al, 2018 ; Hendrix et al, 2016 ; Paraskevakos et al, 2019 ; Lyons et al., 2019 ). For example, the authors of ( Farnes et al, 2018 ) presented a technique for building scalable workflows for analyzing large volumes of satellite imagery data, while Lyons et al (2019) presented a system for analyzing workflows related to weather-sensing data. Other studies have presented generalized methodologies for building scalable workflows for tasks requiring HPC platforms ( Hendrix et al, 2016 ; Castellana et al, 2019 ).…”

Section: Performance Attributesmentioning

confidence: 99%

Data-driven scalable pipeline using national agent-based models for real-time pandemic response and decision support

Bhattacharya

Chen

Hoops

et al. 2022

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

This paper describes an integrated, data-driven operational pipeline based on national agent-based models to support federal and state-level pandemic planning and response. The pipeline consists of ( i) an automatic semantic-aware scheduling method that coordinates jobs across two separate high performance computing systems; ( ii) a data pipeline to collect, integrate and organize national and county-level disaggregated data for initialization and post-simulation analysis; ( iii) a digital twin of national social contact networks made up of 288 Million individuals and 12.6 Billion time-varying interactions covering the US states and DC; ( iv) an extension of a parallel agent-based simulation model to study epidemic dynamics and associated interventions. This pipeline can run 400 replicates of national runs in less than 33 h, and reduces the need for human intervention, resulting in faster turnaround times and higher reliability and accuracy of the results. Scientifically, the work has led to significant advances in real-time epidemic sciences.

show abstract

Section: Performance Attributesmentioning

confidence: 99%

Data-driven scalable pipeline using national agent-based models for real-time pandemic response and decision support

Bhattacharya

Chen

Hoops

et al. 2022

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

show abstract

“…Developing scalable pipelines and workflows for HPC tasks involving large datasets has also been well-studied in literature [14], [21], [25], [29]. For example, the authors of [14] present a technique for building scalable workflows for analyzing large volumes of satellite imagery data, while [25] present a system for analyzing workflows related to weather-sensing data. Other studies have presented generalized methodologies for building scalable workflows for tasks requiring HPC platforms [5], [21].…”

Section: Related Workmentioning

confidence: 99%

Scalable Epidemiological Workflows to Support COVID-19 Planning and Response

Machi

Bhattacharya

Hoops

et al. 2021

2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

View full text Add to dashboard Cite

The COVID-19 global outbreak represents the most significant epidemic event since the 1918 influenza pandemic. Simulations have played a crucial role in supporting COVID-19 planning and response efforts. Developing scalable workflows to provide policymakers quick responses to important questions pertaining to logistics, resource allocation, epidemic forecasts and intervention analysis remains a challenging computational problem. In this work, we present scalable high performance computing-enabled workflows for COVID-19 pandemic planning and response. The scalability of our methodology allows us to run fine-grained simulations daily, and to generate county-level forecasts and other counterfactual analysis for each of the 50 states (and DC), 3140 counties across the USA. Our workflows use a hybrid cloud/cluster system utilizing a combination of local and remote cluster computing facilities, and using over 20,000 CPU cores running for 6-9 hours every day to meet this objective. Our state (Virginia), state hospital network, our university, the DOD and the CDC use our models to guide their COVID-19 planning and response efforts. We began executing these pipelines March 25, 2020, and have delivered and briefed weekly updates to these stakeholders for over 30 weeks without interruption.

show abstract

“…Wide Field Survey Explorer (WISE, Wright et al 2010), which performed a photometric all-sky survey in near-and mid-infrared passbands de-livered over 23 TB 2 (excluding multi-epoch and reject catalogues). Soon, facilities that are currently under development, such as the Square Kilometre Array (SKA; Dewdney et al 2009) and the Large Synoptic Survey Telescope (LSST; Ivezic et al 2008) will provide even larger volumes of data: The LSST is expected to deliver in total 60 PB of raw images 3 , while the SKA would give over 5 ZB (Farnes et al, 2018). Such an exponential growth in the quantity of data has compelled astronomers to develop automatic tools for extracting knowledge about known objects as well as to discover new information.…”

Section: Introductionmentioning

confidence: 99%

Spectroscopic observations of the machine-learning selected anomaly catalogue from the AllWISE Sky Survey

Solarz,

Thomas,

Montenegro-Montes

et al. 2020

Preprint

View full text Add to dashboard Cite

We present the results of a programme to search and identify the nature of unusual sources within the All-sky Wide-field Infrared Survey Explorer (WISE) that is based on a machine-learning algorithm for anomaly detection, namely oneclass support vector machines (OCSVM). Designed to detect sources deviating from a training set composed of known classes, this algorithm was used to create a model for the expected data based on WISE objects with spectroscopic identifications in the Sloan Digital Sky Survey (SDSS). Subsequently, it marked as anomalous those sources whose WISE photometry was shown to be inconsistent with this model. We report the results from optical and near-infrared spectroscopy follow-up observations of a subset of 36 bright (gAB <19.5) objects marked as 'anomalous' by the OCSVM code to verify its performance. Among the observed objects, we identified three main types of sources: i) low redshift (z ∼ 0.03 − 0.15) galaxies containing large amounts of hot dust (53%), including three Wolf-Rayet galaxies; ii) broadline quasi-stellar objects (QSOs) (33%) including low-ionisation broad absorption line (LoBAL) quasars and a rare QSO with strong and narrow ultraviolet iron emission; iii) Galactic objects in dusty phases of their evolution (3%). The nature of four of these objects (11%) remains undetermined due to low signal-to-noise or featureless spectra. The current data show that the algorithm works well at detecting rare but not necessarily unknown objects among the brightest candidates. They mostly represent peculiar sub-types of otherwise well-known sources. To search for even more unusual sources, a more complete and balanced training set should be created after including these rare sub-species of otherwise abundant source classes, such as LoBALs. Such an iterative approach will ideally bring us closer to improving the strategy design for the detection of rarer sources contained within the vast data store of the AllWISE survey.

show abstract

Science Pipelines for the Square Kilometre Array

Cited by 10 publications

References 34 publications

Data-driven scalable pipeline using national agent-based models for real-time pandemic response and decision support

Data-driven scalable pipeline using national agent-based models for real-time pandemic response and decision support

Scalable Epidemiological Workflows to Support COVID-19 Planning and Response

Spectroscopic observations of the machine-learning selected anomaly catalogue from the AllWISE Sky Survey

Contact Info

Product

Resources

About