Leveraging large sensor streams for robust cloud control

Singh, Alok; Stephan, Eric G.; Elsethagen, Todd; Macduff, Matt; Raju, Bibi; Schram, M.; Dam, Kerstin Kleese van; Kerbyson, Darren J.; Altıntaş, Ilkay

doi:10.1109/bigdata.2016.7840839

Cited by 4 publications

(4 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Provenance software can alert developers of anomalous system behavior, but such tools require post hoc analysis, for instance, in the security domain (Pasquier et al, 2018), and for Spark dataflows (Interlandi et al, 2015). Provenance capture is I/O-intensive, although research (Singh et al, 2016) has shown that supervised ML algorithms trained post hoc can alleviate the in situ I/O burden of provenance collection by performing intelligent triage.…”

Section: Provenancementioning

confidence: 99%

Priority research directions for in situ data management: Enabling scientific discovery from diverse data sources

Peterka

Bard

Bennett

et al. 2020

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

In January 2019, the US Department of Energy, Office of Science program in Advanced Scientific Computing Research, convened a workshop to identify priority research directions (PRDs) for in situ data management (ISDM). A fundamental finding of this workshop is that the methodologies used to manage data among a variety of tasks in situ can be used to facilitate scientific discovery from many different data sources—simulation, experiment, and sensors, for example—and that being able to do so at numerous computing scales will benefit real-time decision-making, design optimization, and data-driven scientific discovery. This article describes six PRDs identified by the workshop, which highlight the components and capabilities needed for ISDM to be successful for a wide variety of applications—making ISDM capabilities more pervasive, controllable, composable, and transparent, with a focus on greater coordination with the software stack and a diversity of fundamentally new data algorithms.

show abstract

Section: Provenancementioning

confidence: 99%

Priority research directions for in situ data management: Enabling scientific discovery from diverse data sources

Peterka

Bard

Bennett

et al. 2020

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

show abstract

“…Part of the research focus related to ProvEn has been exploring how traditional provenance analytics (searching and reasoning) can be improved through the use of ML techniques by replacing resource intensive on-demand queries with predictive modeling. In related work applicable to this problem, we have demonstrated training algorithms on empirical metrics from a real instrumented compute cluster, so that it is possible to detect abnormal behavior and trends from streaming metrics and provenance (Singh et al, 2016).…”

Section: Future Directions For Reproducible Researchmentioning

confidence: 99%

Computational reproducibility of scientific workflows at extreme scales

Pouchard

Baldwin

Elsethagen

et al. 2019

The International Journal of High Performance Computing Applica

Self Cite

View full text Add to dashboard Cite

We propose an approach for improved reproducibility that includes capturing and relating provenance characteristics and performance metrics, in a hybrid queriable system, the ProvEn server. The system capabilities are illustrated on two use cases: scientific reproducibility of results in the ACME climate simulations and performance reproducibility in molecular dynamics workflows on HPC computing platforms.

show abstract

“…As shown in Figure 2, ProvEn's service layer enables any analytical environment, e.g., Jupyter notebooks. In [2], we explained how the framework was used for supervised learning to collect and serve observations used as training data. Figure 1.…”

Section: B Proven Frameworkmentioning

confidence: 99%

“…Our past work in modular learning [1] and large sensor networks [2] focuses on using Machine Learning methods to solve workflow performance issues and leverage large sensor networks to achieve robust cloud performance. In continuation of broader scheme of applying Artificial Intelligence Techniques for designing efficient systems, we present our vision to explore the operational data, extract insights and patterns from it and deploy intelligent systems on Belle II that can provide long-term efficiency gains without much intervention.…”

Section: Introductionmentioning

confidence: 99%

Deep Learning on Operational Facility Data Related to Large-Scale Distributed Area Scientific Workflows

Singh

Stephan

Schram

et al. 2017

2017 IEEE 13th International Conference on E-Science (E-Science)

Self Cite

View full text Add to dashboard Cite

Distributed computing platforms provide a robust mechanism to perform large-scale computations by splitting the task and data among multiple locations, possibly located thousands of miles apart geographically. Although such distribution of resources can lead to benefits, it also comes with its associated problems such as rampant duplication of file transfers increasing congestion, long job completion times, unexpected site crashing, suboptimal data transfer rates, unpredictable reliability in a time range, and suboptimal usage of storage elements. In addition, each sub-system becomes a potential failure node that can trigger system wide disruptions. In this vision paper, we outline our approach to leveraging Deep Learning algorithms to discover solutions to unique problems that arise in a system with computational infrastructure that is spread over a wide area. The presented vision, motivated by a real scientific use case from Belle II experiments, is to develop multilayer neural networks to tackle forecasting, anomaly detection and optimization challenges in a complex and distributed data movement environment. Through this vision based on Deep Learning principles, we aim to achieve reduced congestion events, faster file transfer rates, and enhanced site reliability.

show abstract

Leveraging large sensor streams for robust cloud control

Cited by 4 publications

References 26 publications

Priority research directions for in situ data management: Enabling scientific discovery from diverse data sources

Priority research directions for in situ data management: Enabling scientific discovery from diverse data sources

Computational reproducibility of scientific workflows at extreme scales

Deep Learning on Operational Facility Data Related to Large-Scale Distributed Area Scientific Workflows

Contact Info

Product

Resources

About