Hadoop Concepts

Wadkar, Sameer; Siddalingaiah, Madhu

doi:10.1007/978-1-4302-4864-4_2

Cited by 3 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The access date of the last record and the value of the last summation of the difference for the previous records were stored in the AFFDb to be used the next time to calculate the new average with the new record. First, it will be applied if the file exists in the AFFDb using Equation (2). Then the features of the file will be updated in the AFFDb.…”

Section: Aggregated Files Features Databasementioning

confidence: 99%

“…Apache Hadoop is one of the most well-known frameworks in this field [1]. Apache Hadoop is developed to achieve high availability, detect and handle problems, and ensure data consistency [2]. Hadoop consists of two parts: a file system called Hadoop Distributed File System (HDFS) and a programming framework called the Map-Reduce programming model [3], [4].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dynamic Replication Policy on HDFS Based on Machine Learning Clustering

et al. 2023

View full text Add to dashboard Cite

Data growth in recent years has been swift, leading to the emergence of big data science. Distributed File Systems (DFS) are commonly used to handle big data, like Google File System (GFS), Hadoop Distributed File System (HDFS), and others. The DFS should provide the availability of data and reliability of the system in case of failure. The DFS replicates the files in different locations to provide availability and reliability. These replications consume storage space and other resources. The importance of these files differs depending on how frequently they are used in the system. So some of these files do not deserve to replicate many times because it is unimportant in the system. This paper introduces a Dynamic Replication Policy using Machine Learning Clustering (DRPMLC) on HDFS, which uses Machine Learning to cluster the files into different groups and apply other replication policies to each group to reduce the storage consumption, improve the read and write operations time and keep the availability and reliability of HDFS as a High-Performance Distributed Computing (HPDC).

show abstract

Section: Aggregated Files Features Databasementioning

confidence: 99%