Big data analysis using Hadoop cluster

Saldhi, Ankita; Yadav, Dipesh Kumar; Saksena, Dhruv; Goel, Abhinav; Saldhi, Ankur; Indu, S.

doi:10.1109/iccic.2014.7238418

Cited by 13 publications

(5 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A block is the minimum amount of data that can read or write. The default size of HDFS blocks are 128Mb.Files stored in the HDFS are split in to multiple blocks called Chunks that are independent of each other; for example, If the file size is 50 Mb then the HDFS blocks takes only 50Mb of memory space within in the default 128 Mb [36,37,38,39,40,41,42].The name node is responsible for storing the Metadata which means it contains all the information about which shelf the data nodes data are stored. It contains the directory and location of data.…”

Section: Hadoop Distributed File System (Hdfs)mentioning

confidence: 99%

Using Hadoop Technology to Overcome Big Data Problems by Choosing Proposed Cost-efficient Scheduler Algorithm for Heterogeneous Hadoop System (BD3)

Hussein

2020

JSRR

View full text Add to dashboard Cite

Day by day advanced web technologies have led to tremendous growth amount of daily data generated volumes. This mountain of huge and spread data sets leads to phenomenon that called big data which is a collection of massive, heterogeneous, unstructured, enormous and complex data sets. Big Data life cycle could be represented as, Collecting (capture), storing, distribute, manipulating, interpreting, analyzing, investigate and visualizing big data. Traditional techniques as Relational Database Management System (RDBMS) couldn’t handle big data because it has its own limitations, so Advancement in computing architecture is required to handle both the data storage requisites and the weighty processing needed to analyze huge volumes and variety of data economically. There are many technologies manipulating a big data, one of them is hadoop. Hadoop could be understand as an open source spread data processing that is one of the prominent and well known solutions to overcome handling big data problem. Apache Hadoop was based on Google File System and Map Reduce programming paradigm. Through this paper we dived to search for all big data characteristics starting from first three V's that have been extended during time through researches to be more than fifty six V's and making comparisons between researchers to reach to best representation and the precise clarification of all big data V’s characteristics. We highlight the challenges that face big data processing and how to overcome these challenges using Hadoop and its use in processing big data sets as a solution for resolving various problems in a distributed cloud based environment. This paper mainly focuses on different components of hadoop like Hive, Pig, and Hbase, etc. Also we institutes absolute description of Hadoop Pros and cons and improvements to face hadoop problems by choosing proposed Cost-efficient Scheduler Algorithm for heterogeneous Hadoop system.

show abstract

Section: Hadoop Distributed File System (Hdfs)mentioning

confidence: 99%

Using Hadoop Technology to Overcome Big Data Problems by Choosing Proposed Cost-efficient Scheduler Algorithm for Heterogeneous Hadoop System (BD3)

Hussein

2020

JSRR

View full text Add to dashboard Cite

show abstract

“…Ankita Saldi et al [1], focused on statistical data generated in industries. When the generated data is in different formats, environment becomes more challenging to perform functioning.…”

Section: Literature Surveymentioning

confidence: 99%

Text Mining with Apache Hadoop over different Hadoop Clusters Architectures

Lydia¹,

Sekhar²,

Chevuru³

et al. 2019

IJRTE

View full text Add to dashboard Cite

Big data is very much practical for real time applicational systems. One of the mostly used real time application worldwide are on unstructured documents. Large number of documents are managed and maintained through popular leadingBig Data platform is Hadoop. It maintains all the information at Hadoop Distributed File System in Blocks. Irrespective of datasize, BigData has opened its path to store and analyze the data which has consumed time. To overcome this, Hadoophas designed cluster process for large volumes of unstructured data computations. Three different cluster architectures like Standalone, Single node cluster and multi node clusters are considered. In this paper, Big Data allows Hadoop platform to boost the processing speed overlarge datasets through cluster architectures, which are studied and analyzed through text documents from newsgroup20 dataset.It identifies the challenges on text mining and its applications using ApacheHadoop.

show abstract

“…"Big Data is a collection of very large data sets and complex that it becomes difficult to treat using traditional management databases or processing tools application data. The challenges in the areas of capturing, preservation, storage, search, sharing, transfer and analysis, and visualize these [7] data. "The world has been immersed in a sea of data today.…”

Section: Clien Servementioning

confidence: 99%