A cloud service architecture for analyzing big monitoring data

Abstract

Cloud monitoring is of a source of big data that are constantly produced from traces of infrastructures, platforms, and applications. Analysis of monitoring data delivers insights of the system's workload and usage pattern and ensures workloads are operating at optimum levels. The analysis process involves data query and extraction, data analysis, and result visualization. Since the volume of monitoring data is big, these operations require a scalable and reliable architecture to extract, aggregate, and analyze data in an arbitrary range of granularity. Ultimately, the results of analysis become the knowledge of the system and should be shared and communicated. This paper presents our cloud service architecture that explores a search cluster for data indexing and query. We develop REST APIs that the data can be accessed by different analysis modules. This architecture enables extensions to integrate with software frameworks of both batch processing (such as Hadoop) and stream processing (such as Spark) of big data. The analysis results are structured in Semantic Media Wiki pages in the context of the monitoring data source and the analysis process. This cloud architecture is empirically assessed to evaluate its responsiveness when processing a large set of data records under node failures.

Big Data Meet Cyber-Physical Systems: A Panoramic Survey

“…Cloud-based analysis in CPS constitutes a scalable and reliable architecture to perform analytics operations on big data stream, such as extracting, aggregating and analyzing data of different granularities [180]. A massive amount of data are usually stored in spreadsheets or other applications, and a cloud-based analytics service, using statistical analysis and machine learning, helps reduce the big data to a manageable size so information can be extracted, hypothesis can be tested, and conclusions can be drawn from non-numerical data such as photos.…”

Brain Imaging Big Data Mining and Fusion Method Based on Cognitive Intelligence

“…The more ants passing through the path, the larger the amount of information, the easier the path is to be selected and the smaller the amount of information. In the collaborative phase, the exchange of information between the candidate solutions is expected to produce a better performance solution, similar to the learning mechanism of the learning automaton [10,11]. Ant colony algorithm is actually a class of multi-subject system.…”

“…ijrepresents the increment of the amount of information on the path between the nodes, and its value is determined as shown in equation (10). 0;( , ) ( , )…”

Big Data Storage and Parallel Analysis of Grid Equipment Monitoring System

“…D. W. Wang et al [8] described a solution of intelligent grid condition monitoring cloud platform based on Hadoop technology, but they were unable to present an actual prototype system. S. Singh et al [6] designed and implemented a cloud-based service architecture for power monitoring data, but they did not provide the methods of data analysis and distributed storage model. D. S. Yang et al [10] discussed and analyzed the key technologies of high-speed storage and retrieval of the massive power data, but they did not give an actual model that has been implemented.…”