Real-time data-intensive computing

Parkinson, Dilworth Y.; Beattie, K.; Chen, Xian; Correa, Joaquin; Dart, Eli; Daurer, Benedikt J.; Deslippe, Jack; Hexemer, Alexander; Krishnan, Harinarayan; MacDowell, Alastair A.; Maia, Filipe R. N. C.; Marchesini, Stefano; Padmore, H. A.; Patton, S.; Perciano, Talita; Sethian, James A.; Shapiro, David A.; Stromsness, Rune; Tamura, Nobumichi; Tierney, Brian; Tull, C. E.; Ushizima, Daniela

doi:10.1063/1.4952921

Cited by 11 publications

(3 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, we also see that such a framework results in beamline scientists and users not having complete control over computational resources which are subject to facility timetabling, downtimes, and regulations. Since real‐time feedback is crucial for some experiments, the inevitable computational queuing times from the use of the large computational resources can result in serious limitations for such users …”

Section: Scientific Community‐driven Big Data Approachesmentioning

confidence: 99%

“…In recent years, a vision of a “superfacility” has been proposed by the Advanced Light Source (ALS) and the National Energy Research Scientific Computing Center (NERSC) at Lawrence Berkeley National Laboratory (LBNL) in California, which is meant to help users focus only on those scientific meaningful data from the ALS which could lead to new discoveries, rather than have to deal with an unstructured massive amount of raw data . Such an approach is achievable only by giving users real‐time simultaneous access to the experimental, computational, and algorithmic resources at synchrotrons.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Synchrotron Big Data Science

Wang

Steiner

Sepe

2018

Small

View full text Add to dashboard Cite

The rapid development of synchrotrons has massively increased the speed at which experiments can be performed, while new techniques have increased the amount of raw data collected during each experiment. While this has created enormous new opportunities, it has also created tremendous challenges for national facilities and users. With the huge increase in data volume, the manual analysis of data is no longer possible. As a result, only a fraction of the data collected during the time- and money-expensive synchrotron beam-time is analyzed and used to deliver new science. Additionally, the lack of an appropriate data analysis environment limits the realization of experiments that generate a large amount of data in a very short period of time. The current lack of automated data analysis pipelines prevents the fine-tuning of beam-time experiments, further reducing their potential usage. These effects, collectively known as the "data deluge," affect synchrotrons in several different ways including fast data collection, available local storage, data management systems, and curation of the data. This review highlights the Big Data strategies adopted nowadays at synchrotrons, documenting this novel and promising hybridization between science and technology, which promise a dramatic increase in the number of scientific discoveries.

show abstract

Section: Scientific Community‐driven Big Data Approachesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Synchrotron Big Data Science

Wang

Steiner

Sepe

2018

Small

View full text Add to dashboard Cite

show abstract

“…The performance of transfers between storage and the widearea network (WAN) have also been studied with increasing enthusiasm as networking technologies and methodologies [20] have advanced to the point where geographically distributed workflows and large, public datasets are now enabling new scientific discovery [2], [3], [21]. Globus, GridFTP-based transfer tools, and bbcp are ubiquitous high-performance data transfer tools used to this end [1]- [3], and characterizing the ways in which Globus and GridFTP are used in multisite workflows and distributed HPC environments is the subject of growing interest [12], [22], [23]. As with the efforts to characterize user interactions with archival storage systems, however, studies of wide-area data transfers have explored only the storage-WAN and WAN-storage components of data transfer.…”

Section: Introductionmentioning

confidence: 99%

Understanding Data Motion in the Modern HPC Data Center

Lockwood¹,

Snyder²,

Byna³

et al. 2019

2019 IEEE/ACM Fourth International Parallel Data Systems Workshop (PDSW)

View full text Add to dashboard Cite

The utilization and performance of storage, compute, and network resources within HPC data centers have been studied extensively, but much less work has gone toward characterizing how these resources are used in conjunction to solve larger scientific challenges. To address this gap, we present our work in characterizing workloads and workflows at a data-center-wide level by examining all data transfers that occurred between storage, compute, and the external network at the National Energy Research Scientific Computing Center over a three-month period in 2019. Using a simple abstract representation of data transfers, we analyze over 100 million transfer logs from Darshan, HPSS user interfaces, and Globus to quantify the load on data paths between compute, storage, and the wide-area network based on transfer direction, user, transfer tool, source, destination, and time. We show that parallel I/O from user jobs, while undeniably important, is only one of several major I/O workloads that occurs throughout the execution of scientific workflows. We also show that this approach can be used to connect anomalous data traffic to specific users and file access patterns, and we construct time-resolved user transfer traces to demonstrate that one can systematically identify coupled data motion for individual workflows.

show abstract

Synchrotron radiation data-driven artificial intelligence approaches in materials discovery

Li,

Xing,

et al. 2024

Artificial Intelligence Chemistry

View full text Add to dashboard Cite

Real-time data-intensive computing

Cited by 11 publications

References 12 publications

Synchrotron Big Data Science

Synchrotron Big Data Science

Understanding Data Motion in the Modern HPC Data Center

Synchrotron radiation data-driven artificial intelligence approaches in materials discovery

Contact Info

Product

Resources

About