Storage Mining: Where IT Management Meets Big Data Analytics

“…The popular R programming language supports a wide range of statistical methods, as well as machine learning operations such as linear and logical regressions, classifications and clustering [10]. To run programs written in R, Hadoop and Spark frameworks have the correspondent additional software like RHadoop [11] and SparkR [12].…”

Section: Overview Of Frameworkmentioning

confidence: 99%

Performance evaluation of distributed computing environments with Hadoop and Spark frameworks

Taran

Alienin

Stirenko

et al. 2017

2017 IEEE International Young Scientists Forum on Applied Physics and Engineering (YSF)

View full text Add to dashboard Cite

Abstract-Recently, due to rapid development of information and communication technologies, the data are created and consumed in the avalanche way. Distributed computing create preconditions for analyzing and processing such Big Data by distributing the computations among a number of compute nodes. In this work, performance of distributed computing environments on the basis of Hadoop and Spark frameworks is estimated for real and virtual versions of clusters. As a test task, we chose the classic use case of word counting in texts of various sizes. It was found that the running times grow very fast with the dataset size and faster than a power function even. As to the real and virtual versions of cluster implementations, this tendency is the similar for both Hadoop and Spark frameworks. Moreover, speedup values decrease significantly with the growth of dataset size, especially for virtual version of cluster configuration. The problem of growing data generated by IoT and multimodal (visual, sound, tactile, neuro and brain-computing, muscle and eye tracking, etc.) interaction channels is presented. In the context of this problem, the current observations as to the running times and speedup on Hadoop and Spark frameworks in real and virtual cluster configurations can be very useful for the proper scaling-up and efficient job management, especially for machine learning and Deep Learning applications, where Big Data are widely present.

show abstract

“…Data can be provided as a service by using big data in Cloud Computing [8]. Hadoop can be used for big data management [9]. Different types of analysis on the big data hadoop are done.…”

Section: Related Workmentioning

confidence: 99%

A Time Based Analysis of Data Processing on Hadoop Cluster

Pal

Agrawal

2014

2014 International Conference on Computational Intelligence and Communication Networks

View full text Add to dashboard Cite

Data when it becomes in that much amount that it cannot be managed by the traditional database management system then it is Bigdata. It is difficult to manage this much amount of the data. Hadoop is a technological answer to the Big Data. Data storage and retrieval of information from the data is done by the Hadoop Distributed File System and the Map Reduce Programming model. MapReduce provides effective bench marks for retrieving the information from the Big Data. In this paper we present our experimental work done on the Hadoop Cluster. We have analyzed the time required by the cluster for processing the data with increasing number of nodes into the cluster. We started with a single node and then increase the node by one each time. We have analyzed three types of time. The real time, user time, system time is analyzed.

show abstract

Storage Mining: Where IT Management Meets Big Data Analytics

Cited by 22 publications

References 5 publications

The overview of big data storage and management

The overview of big data storage and management

Performance evaluation of distributed computing environments with Hadoop and Spark frameworks

A Time Based Analysis of Data Processing on Hadoop Cluster

Contact Info

Product

Resources

About