A Performance Analysis of MapReduce Task with Large Number of Files Dataset in Big Data Using Hadoop

Pal, Amrit; Jain, Kunal; Agrawal, Pinki; Agrawal, Sanjay

doi:10.1109/csnt.2014.124

Cited by 14 publications

(2 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…According to the words of Amrit Pal, Kunal Jain, Pinki Agrawal, Sanjay Agrawal [3] log is the main source of system operation status, user behavior and systems actions. Log analyses helps to improve the business strategies as well as to generate statistical report.…”

Section: Nadeem Akhtar Mohd Vasim Ahamad Shahbaaz Ahmad[2]mentioning

confidence: 99%

Big Data Analysis of Fetched Logs using Hadoop Framework

Chiddarwar¹,

Chhajed²,

Deshmukh³

et al. 2017

International Journal of Advanced Research in Computer and Comm

View full text Add to dashboard Cite

With every second slipping away, there are thousands of activities carried out on the internet. The big data generated out of logs is very crucial and important to provide a superior backbone to the whole IT infrastructure. It is not an easy task to analyse the logs generated by system as there are numbers of discrete type machines available in a network. The growing size and complexity of log files has made the manual analysis of system logs by administrators prohibitive. This fact makes it important for tools and techniques that will allow some form of automation in management and analysis of system logs to be developed. The proposed technique uses Hadoop framework to process huge amount of data. To extract useful information data mining technique is used, among which clustering is most popular. Various types of clustering like Connectivity-based clustering, Centroid-based clustering (K-Means Clustering), Density-based clustering, etc. can be used to cluster the log files. Centroid-based (K-Means Clustering) is the most effective technique amongst all. The input logs will be clustered using K-Means clustering and then processed in Hadoop framework. The analysis result can be used for detecting security threats in network and inform the administrator about the Intrusion Detection, SQL-Injection, system error, etc.

show abstract

Section: Nadeem Akhtar Mohd Vasim Ahamad Shahbaaz Ahmad[2]mentioning

confidence: 99%

Big Data Analysis of Fetched Logs using Hadoop Framework

Chiddarwar¹,

Chhajed²,

Deshmukh³

et al. 2017

International Journal of Advanced Research in Computer and Comm

View full text Add to dashboard Cite

show abstract

“…M. Moorthy et al, [3], presented in their work, the data clustering will be handled under distributed Hadoop environment which serves choice in crop planning by forecast the demand in the market at the earliest. A. Pal et al, [4] proposed a popular Map-Reduce concept utilized clustered file system extensively with Hadoop Distributed File System (HDFS). K. Grolinger et al, [5] presented the purpose behind the Map-Reduce paradigm is high scalable which executes massively parallel and distributed over a huge number of computing nodes.…”

Section: Introductionmentioning

confidence: 99%