LOGAIDER: A Tool for Mining Potential Correlations of HPC Log Events

Di, Sheng; Gupta, Rinku; Snir, Marc; Pershey, Eric; Cappello, Franck

doi:10.1109/ccgrid.2017.18

Cited by 43 publications

(21 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It can not be subsumed by temporal characteristics such as MTBF or temporal recurrence. Interestingly, researchers at LLNL and Argonne National Laboratory have independently veriied that our observations about spatial distribution of failures hold true in their systems as well [6,30], which are not Cray systems and have very diferent system composition.…”

Section: Discussionmentioning

confidence: 62%

“…The development of such analysis methodologies, tools and predictive models can lead to deeper insights about the underlying system behavior but it remains too complex to allow system administrators and users to compare reliability of two diferent systems. This work has introduced two new metrics to characterize temporal and spatial properties of failures that can be used by other failure analysis eforts [2,6,30].…”

Section: Related Work and Conclusionmentioning

confidence: 99%

“…But, it is our continued efort and inal goal to make parts of the data available for researchers in near future. Our study will enable others to learn from these lessons, apply new metrics, and test the indings presented here in their environment [6,30]. We hope that these indings could be potentially applied to novel systems such as Minerva [21].…”

Section: Related Work and Conclusionmentioning

confidence: 99%

See 2 more Smart Citations

Failures in large scale systems

Gupta

Patel

Engelmann

et al. 2017

Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

105

View full text Add to dashboard Cite

Resilience is one of the key challenges in maintaining high eiciency of future extreme scale supercomputers. Researchers and system practitioners rely on ield-data studies to understand reliability characteristics and plan for future HPC systems. In this work, we compare and contrast the reliability characteristics of multiple largescale HPC production systems. Our study covers more than one billion compute node hours across ive diferent systems over a period of 8 years. We conirm previous indings which continue to be valid, discover new indings, and discuss their implications.

show abstract

Section: Discussionmentioning

confidence: 62%

Section: Related Work and Conclusionmentioning

confidence: 99%

Section: Related Work and Conclusionmentioning

confidence: 99%

See 1 more Smart Citation

Failures in large scale systems

Gupta

Patel

Engelmann

et al. 2017

Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

105

View full text Add to dashboard Cite

show abstract

“… Balliu et al (2015) propose B I DA L , a tool to characterize the workload of cloud infrastructures, They use log data from Google data clusters for evaluation and incorporate support to popular analysis languages and storage backends on their tool. Di et al (2017) propose L OG A IDER , a tool that integrates log mining and visualization to analyze different types of correlation (e.g., spatial and temporal). In this study, they use log data from Mira, an IBM Blue Gene-based supercomputer for scientific computing, and reported high accuracy and precision in uncovering correlations associated with failures.…”

Section: Resultsmentioning

confidence: 99%

Log-based software monitoring: a systematic mapping study

Cândido

Aniche

Deursen

2021

PeerJ Computer Science

View full text Add to dashboard Cite

Modern software development and operations rely on monitoring to understand how systems behave in production. The data provided by application logs and runtime environment are essential to detect and diagnose undesired behavior and improve system reliability. However, despite the rich ecosystem around industry-ready log solutions, monitoring complex systems and getting insights from log data remains a challenge. Researchers and practitioners have been actively working to address several challenges related to logs, e.g., how to effectively provide better tooling support for logging decisions to developers, how to effectively process and store log data, and how to extract insights from log data. A holistic view of the research effort on logging practices and automated log analysis is key to provide directions and disseminate the state-of-the-art for technology transfer. In this paper, we study 108 papers (72 research track papers, 24 journals, and 12 industry track papers) from different communities (e.g., machine learning, software engineering, and systems) and structure the research field in light of the life-cycle of log data. Our analysis shows that (1) logging is challenging not only in open-source projects but also in industry, (2) machine learning is a promising approach to enable a contextual analysis of source code for log recommendation but further investigation is required to assess the usability of those tools in practice, (3) few studies approached efficient persistence of log data, and (4) there are open opportunities to analyze application logs and to evaluate state-of-the-art log analysis techniques in a DevOps context.

show abstract

“…In the Mira cluster with diverse system logs, the Reliability, Availability, and Serviceability (RAS) log is our focus. The [12].…”

Section: Blue Gene/q Mira Cluster and Ras Logsmentioning

confidence: 99%