Performance Analysis of Small Files in HDFS using Clustering Small Files based on Centroid Algorithm

Rathidevi, R.; Parameswari, R.

doi:10.1109/i-smac49090.2020.9243418

Cited by 5 publications

(3 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…N. Alange et al [26] compared various existing techniques for small file problem based on the performance throughput. The CSFC technique, developed by R. Rathidevi et al [27], involves grouping together small files that are connected in some way. HDFS is used for additional processing of the merged small files after clustering.…”

Section: Related Workmentioning

confidence: 99%

A Dynamic Repository Approach for Small File Management With Fast Access Time on Hadoop Cluster: Hash Based Extended Hadoop Archive

Sharma

Barwar

Afthanorhan³

et al. 2022

IEEE Access

View full text Add to dashboard Cite

Small file processing in Hadoop is one of the challenging task. The performance of the Hadoop is quite good when dealing with large files, because they require lesser meta data and consume less memory. But while dealing with enormous amount of small files, meta data grows linearly and Name Node memory got overloaded hence overall performance of the Hadoop degrades. This paper presents a dual merge technique HB-EHA (Hash Based-Extended Hadoop Archive) that will resolve the small file issue of Hadoop and provide an excellent solution for massive small files that are generated in the health care management applications. The proposed technique merges the small files using two level compaction, therefore size of meta data at name node get reduced and less memory will be used. The indexing will be carried out over the archives and files can be accessed after merging in real time. Index files in the proposed approach can read partially that improves the name node memory usage and also offers the file appending capability in the existing archive. The proposed technique first creates Hadoop archive from the small files and then uses two special hash functions i.e. SSHF (Scalable-Splittable Hash Function) and HT-MMPHF (Hollow Trie Monotone Minimal Perfect Hash Function), SSHF is used to dynamically distribute the archives meta-data to the associated slave index files, and these slave index files will be further written to the final index files, the order of the meta-data in final index file will be preserved by the HT-MMPHF. The evaluation outcome exhibit that proposed technique is 13% & 17% faster than HDFS with caching enabled and disabled respectively, and 38% & 47% faster than the HAR with caching and without caching respectively. While comparing with map file, proposed technique is 28 & 35 time faster with caching and without caching respectively. HB-EHA is maximum 40% & 28% faster than the HBAF with and without caching respectively.

show abstract

Section: Related Workmentioning

confidence: 99%

A Dynamic Repository Approach for Small File Management With Fast Access Time on Hadoop Cluster: Hash Based Extended Hadoop Archive

Sharma

Barwar

Afthanorhan³

et al. 2022

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Li et al [21] proposed a virtual file pool technology, which uses a memory-mapped file method to achieve an efficient data reading process and provides a data exchange strategy in the pool. Rathidevi and Parameswari [22] proposed a centroid-based clustering of the small files method, which placed related files in clusters to improve data access performance. Xiang et al [23] proposed a Hadoop distributed file system storage strategy based on the response time of data nodes, and the simulation results show that this strategy can realize the distributed storage of big data and avoid the emergence of hot nodes.…”

Section: Medical Big Data Storage and Accessmentioning

confidence: 99%

Hadoop-Based Medical Image Storage and Access Method for Examination Series

Huang

Wang³

et al. 2021

Mathematical Problems in Engineering

View full text Add to dashboard Cite

Under the background of electronic medical data, doctors use electronic images to replace the traditional film for diagnosis, and patients can view examination images at any time through various electronic means. The storage and frequent reading of massive data bring new challenges. Given the characteristics of the size and quantity of image files generated by different examination types, different merging strategies are proposed to improve the storage performance of the files; according to the characteristics of medical data with examination as the basic unit, a two-level model combined with medical imaging information is proposed. The indexing mechanism solves the problem that SEQ files cannot be read randomly without an index; given the time characteristics of data access, an improved 2Q algorithm is proposed to cache the prefetched files and the read files in different cache queues, which improves the efficiency of file reading. In the experimental comparison, the proposed algorithm surpasses the baseline method in storage and access performance.

show abstract

“…The authors have also given an idea about the tools used for Big Data storage and processing along with the algorithms used in the Big Data processing. Hadoop, HDFS [3], CDH [4], Mon-goDB [5], Apache spark [6], Apache Solr [7], [8],Alteryx [9] Designer, Data Meer [9], Google Big Query [9] etc are the tools used to store process and analyze the Big Data and Support Vector Machine [10], Neural Network, Logistic regression [11], Linear Regression [11], Nearest Neighbor, Decision tree, Naive Bayes are the Algorithms used in healthcare analysis.…”

Section: Introductionmentioning

confidence: 99%

A Survey of Big Data Tools for Remote Healthcare Services

Ray,

Mainuddin,

Singha

et al. 2023

Preprint

View full text Add to dashboard Cite

Health Data storage and analytics are the two important parts of the health or drugs business and health research. Health Data storage and management are important for storing and retrieving Health Data efficiently. The scalable and flexible infrastructure capable of efficient storing and processing the millions of Health Data generated every day must be very much needed for the industry. Big Data tools and NoSQL databases are becoming the necessary tools to store millions of data and respond to given queries efficiently. In general relational databases are used for many applications, but they are not at all efficient for storing and processing Big Data. The drawbacks of relational databases motivate the developers to use NoSQL databases and Big Data tools while creating the Health application. A comparative study must be made to find out the performance of different Big Data platforms along with the NoSQL Databases to store Health Data for a Health-related application and to respond to queries efficiently. In this paper, we have tested the performance of Hadoop, HIVE, Cassandra, MongoDB, HBASE and NGMR to respond to different queries to store Health Data for a remote health framework. Such a study helps us to come up with new ideas that can help us to store and retrieve the Health Data of a particular application using a data model that is iteratively tested and modified. It provides us with an understanding of the Big Data solution or tool as well. The key aim of the study is to make a review on the capability of data storage and the read/write performance of the tools

show abstract

Performance Analysis of Small Files in HDFS using Clustering Small Files based on Centroid Algorithm

Cited by 5 publications

References 9 publications

A Dynamic Repository Approach for Small File Management With Fast Access Time on Hadoop Cluster: Hash Based Extended Hadoop Archive

A Dynamic Repository Approach for Small File Management With Fast Access Time on Hadoop Cluster: Hash Based Extended Hadoop Archive

Hadoop-Based Medical Image Storage and Access Method for Examination Series

A Survey of Big Data Tools for Remote Healthcare Services

Contact Info

Product

Resources

About