Small Sized File Storage Problems in Hadoop Distributed File System

Alange, Neeta; Mathur, Anjali

doi:10.1109/icssit46314.2019.8987739

Cited by 6 publications

(2 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…HDFS is the underlying foundation of Hadoop to store massive rural tourism data. HDFS sliced the rural tourism historical data into chunks and kept them in different rural tourism servers ( Alange & Mathur, 2019 ).…”

Section: Key Technologiesmentioning

confidence: 99%

Construction of a rural tourism information service management system for multi-source heterogeneous data processing

Huang

2023

PeerJ Computer Science

View full text Add to dashboard Cite

This study offers an integrated service management system for rural tourist information based on a cloud platform to address the three main issues of high platform concurrency, difficulty storing and managing data, and trouble sharing data functions. Three levels—data, process, and architecture—are considered in the analysis and design of the platform. The Hadoop data storage system makes possible the collection, storage, administration, and exchange of data functions for large amounts of heterogeneous data from many different sources by utilising Netty data transmission technology, hybrid data storage technology, and the Web Foundation. The results demonstrate that the system’s response time is low, and the CPU consumption time and the average utilisation rate meet the actual needs. They resolve issues with the current rural tourism platforms application, such as the difficulty of data collection, the low rate of reuse, the low rate of sharing, the lack of timely updates, and severe island phenomena.

show abstract

Section: Key Technologiesmentioning

confidence: 99%

Construction of a rural tourism information service management system for multi-source heterogeneous data processing

Huang

2023

PeerJ Computer Science

View full text Add to dashboard Cite

show abstract

“…Sharma V.S et al [25] evaluated various existing techniques in Hadoop that will provide the solution to the small file problem of the HDFS. N. Alange et al [26] compared various existing techniques for small file problem based on the performance throughput. The CSFC technique, developed by R. Rathidevi et al [27], involves grouping together small files that are connected in some way.…”

Section: Related Workmentioning

confidence: 99%

A Dynamic Repository Approach for Small File Management With Fast Access Time on Hadoop Cluster: Hash Based Extended Hadoop Archive

Sharma

Barwar

Afthanorhan³

et al. 2022

IEEE Access

View full text Add to dashboard Cite

Small file processing in Hadoop is one of the challenging task. The performance of the Hadoop is quite good when dealing with large files, because they require lesser meta data and consume less memory. But while dealing with enormous amount of small files, meta data grows linearly and Name Node memory got overloaded hence overall performance of the Hadoop degrades. This paper presents a dual merge technique HB-EHA (Hash Based-Extended Hadoop Archive) that will resolve the small file issue of Hadoop and provide an excellent solution for massive small files that are generated in the health care management applications. The proposed technique merges the small files using two level compaction, therefore size of meta data at name node get reduced and less memory will be used. The indexing will be carried out over the archives and files can be accessed after merging in real time. Index files in the proposed approach can read partially that improves the name node memory usage and also offers the file appending capability in the existing archive. The proposed technique first creates Hadoop archive from the small files and then uses two special hash functions i.e. SSHF (Scalable-Splittable Hash Function) and HT-MMPHF (Hollow Trie Monotone Minimal Perfect Hash Function), SSHF is used to dynamically distribute the archives meta-data to the associated slave index files, and these slave index files will be further written to the final index files, the order of the meta-data in final index file will be preserved by the HT-MMPHF. The evaluation outcome exhibit that proposed technique is 13% & 17% faster than HDFS with caching enabled and disabled respectively, and 38% & 47% faster than the HAR with caching and without caching respectively. While comparing with map file, proposed technique is 28 & 35 time faster with caching and without caching respectively. HB-EHA is maximum 40% & 28% faster than the HBAF with and without caching respectively.

show abstract