Monitoring of IaaS and scientific applications on the Cloud using the Elasticsearch ecosystem

Bagnasco, S.; Berzano, D.; Guarise, Andrea; Lusso, S.; Masera, M.; Vallero, S.

doi:10.1088/1742-6596/608/1/012016

Cited by 31 publications

(12 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…These tools can construct a complete chain that providing functionalities from log collecting, data storing and indexing to data visualizing. There are many works that use the ELK combination to achieve the goal of data processing [19][20] [21]. However, ELK only provides a basic data processing workflow, and it still need system maintainers to make effort on further analyses.…”

Section: Pos(isgc2019)033mentioning

confidence: 99%

A Blueprint of Log Based Monitoring and Diagnosing Framework in Large Distributed Environments

Zhao¹,

Wang²,

Xiao³

et al. 2019

Proceedings of International Symposium on Grids &Amp; Clouds 2019 — PoS(ISGC2019)

View full text Add to dashboard Cite

Distributed systems have kept scaling upward since this concept appears, and they soon evolve to environments that contain heterogeneous components playing different roles, making it difficult to understand how the large environment works or if any undesired matters happened from security point of view. Logs, produced by devices, subsystems and running processes, are a very important source to help system maintainers to get relative security knowledge. But there are too many logs and too many kinds of logs to deal with, which makes manual checking impossible. In this work we will share some of our experiences in log processing and analyzing. We have summarized some common major steps that appear in most of the existing log analysis approaches, including log selection, log classification, information analyses and result feedback. We also represent a general framework that monitors events, analyzes hidden information and diagnoses the healthy state for large distributed computing environments bases on logs. Although the framework we initially designed was for the maintenance for CNGrid, its process is adaptable to other distributed computing environments.

show abstract

Section: Pos(isgc2019)033mentioning

confidence: 99%

A Blueprint of Log Based Monitoring and Diagnosing Framework in Large Distributed Environments

Zhao¹,

Wang²,

Xiao³

et al. 2019

Proceedings of International Symposium on Grids &Amp; Clouds 2019 — PoS(ISGC2019)

View full text Add to dashboard Cite

show abstract

“…This step can be achieved after the fact that very good results. All business groups will require an exception panel, for example, payment system business groups; it was probably about 10 different small services [4].…”

Section: Evolution Of Our Log Analysis Systemmentioning

confidence: 99%

Research on Establish an Efficient Log Analysis System with Kafka and Elastic Search

Wei¹,

Manman²,

Xu³

2017

JSEA

View full text Add to dashboard Cite

Search logs in a timely and efficient manner are an important part of SRE (Site Reliability Engineer). Logs help us solve the problems during our development work. In this paper, we will introduce you a way how to build an efficient logs analysis system based on kafka and Elastic Search. We hope you can learn something through the iteration of the Version and get some inspiration with your own log analysis system.

show abstract

“…Elasticearch -Query Engine Elasticsearch(ES), is a open source searching and analytical engine based on Lucene, and supports cloud service [3]. Through it we can dig deeper into the data easily by amplifying and narrowing the range of search and analysis, most important is real-time [4].…”

Section: Logstash -Log Pipelinementioning

confidence: 99%

Log Real-time Management Scheme Based on LEK

Lei¹,

Wang²,

He³

2015

Proceedings of the 2015 2nd International Workshop on Materials Engineering and Computer Sciences

View full text Add to dashboard Cite

The growing importance of log analysis and the management play an important role in the era of big data especially. With mass log information increasing, it's difficult to extract the needed information quickly, What's more, most cases are related information. Logstash can solve this problem suitably because of a built-in logging service.It can be combined with help from many log slaves to find data easily. Strictly speaking, Logstash alone are unable to rationalize and centralized the log files. This article integrated Logstash, Elasticsearch and Kibana into an organic whole, which is applied to an insurance business trading system LEK. It implements collection, query, polymerization and visual function of the transaction log, and form the unified convenient log management and analysis tool.

show abstract

Monitoring of IaaS and scientific applications on the Cloud using the Elasticsearch ecosystem

Cited by 31 publications

References 4 publications

A Blueprint of Log Based Monitoring and Diagnosing Framework in Large Distributed Environments

A Blueprint of Log Based Monitoring and Diagnosing Framework in Large Distributed Environments

Research on Establish an Efficient Log Analysis System with Kafka and Elastic Search

Log Real-time Management Scheme Based on LEK

Contact Info

Product

Resources

About