Performance Evaluation of Merging Techniques for Handling Small Size Files in HDFS

Sharma, Vijay Shankar; Barwar, N. C.

doi:10.1007/978-981-15-8335-3_13

Cited by 4 publications

(4 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This paper provides quick insight into the possible solution to the small file problem. Sharma V.S et al [25] evaluated various existing techniques in Hadoop that will provide the solution to the small file problem of the HDFS. N. Alange et al [26] compared various existing techniques for small file problem based on the performance throughput.…”

Section: Related Workmentioning

confidence: 99%

A Dynamic Repository Approach for Small File Management With Fast Access Time on Hadoop Cluster: Hash Based Extended Hadoop Archive

Sharma

Barwar

Afthanorhan³

et al. 2022

IEEE Access

Self Cite

View full text Add to dashboard Cite

Small file processing in Hadoop is one of the challenging task. The performance of the Hadoop is quite good when dealing with large files, because they require lesser meta data and consume less memory. But while dealing with enormous amount of small files, meta data grows linearly and Name Node memory got overloaded hence overall performance of the Hadoop degrades. This paper presents a dual merge technique HB-EHA (Hash Based-Extended Hadoop Archive) that will resolve the small file issue of Hadoop and provide an excellent solution for massive small files that are generated in the health care management applications. The proposed technique merges the small files using two level compaction, therefore size of meta data at name node get reduced and less memory will be used. The indexing will be carried out over the archives and files can be accessed after merging in real time. Index files in the proposed approach can read partially that improves the name node memory usage and also offers the file appending capability in the existing archive. The proposed technique first creates Hadoop archive from the small files and then uses two special hash functions i.e. SSHF (Scalable-Splittable Hash Function) and HT-MMPHF (Hollow Trie Monotone Minimal Perfect Hash Function), SSHF is used to dynamically distribute the archives meta-data to the associated slave index files, and these slave index files will be further written to the final index files, the order of the meta-data in final index file will be preserved by the HT-MMPHF. The evaluation outcome exhibit that proposed technique is 13% & 17% faster than HDFS with caching enabled and disabled respectively, and 38% & 47% faster than the HAR with caching and without caching respectively. While comparing with map file, proposed technique is 28 & 35 time faster with caching and without caching respectively. HB-EHA is maximum 40% & 28% faster than the HBAF with and without caching respectively.

show abstract

Section: Related Workmentioning

confidence: 99%

A Dynamic Repository Approach for Small File Management With Fast Access Time on Hadoop Cluster: Hash Based Extended Hadoop Archive

Sharma

Barwar

Afthanorhan³

et al. 2022

IEEE Access

Self Cite

View full text Add to dashboard Cite

show abstract

“…The system achieved accuracy of 98.88% with AUC value of 99.93% for task1. It had obtained an accuracy of 93.03% for task 2 using transfer learning mechanism.The authors proposed a solution for dealing with small files, the concept can be used in implementing database with Hadoop and can provide a simple way to store health data [22,24,23].…”

Section: Introductionmentioning

confidence: 99%

Ocular Disease Severity Identification and Performance Optimisation using Custom Net Model

Bhakar,

Vishnawat,

Kundu

et al. 2024

SCPE

View full text Add to dashboard Cite

Early detection and timely cure of ocular disease play a vital role to avoid irreversible vision issues in daily life. The technique fundus assessment utilizes color fundus photography, which is a very effective tool though it is expensive. Since rare symptoms of the disease are detected at the initial stage of the disease, still automated and optimized models are in urgent need for the detection of the ocular disease. Additionally, existing systems focus on image-level detection for the treatment of eyes without association employing the left and right eye information. Although they concentrate only on one or two features of the ocular disease at a time. Taking into consideration severity detection and multilabel categorization plays a vital role in ocular disease detection. So, we develop a framework to detect the disease in the early phase. And then apply the classification model for the multilabel classification of the disease. our proposed experimental result proves that the proposed Custom net model provides 99.15% of accuracy compared to the existing baseline model such as Vgg16, 19, Resnet-50 and Inception V3. The performance optimization of the proposed model is evaluated on the public datasets.

show abstract

“…The system achieves 75.2% accuracy after the process of validation. The authors proposed a solution for dealing with small files, the concept can be used in implementing database with Hadoop and can provide a simple way to store health data [28,30,29]. applied to get 144 features.…”

Section: Introductionmentioning

confidence: 99%

A Hybrid Model: Random Classification and Feature Selection Approach for Diagnosis of the Parkinson Syndrome

Bhakar,

Shekhawat,

Kundu

et al. 2024

SCPE

View full text Add to dashboard Cite

Nowadays Parkinson’s disease has been discovered that approximately 94% of people suffer from voice disorder problems. A neurodegenerative can identify PD patients through examination and multiple scanning tests. So, it usually takes more time to diagnose the disease at the early stage. Current work has identified that speech disorders can be a significant signal for Parkinson’s disease. Therefore, this work proposed a fusion model to identify the speech disorder at the starting stage of the disease. In this process, the author has tested a model with a different pattern of feature selection method as well as classification mode and created a system with the best pattern. For the creation of pattern, three types of feature selection methods namely Chi-square, genetic algorithm and Embedded random forest method and four classifier models such as KNN, Naïve Bayes, SVM, Decision tree and Random Forest have been utilized. To analyze the performance of the system speech public dataset from the UCI repository,the authors applied the combination of the Embedded random feature selection method and random forest classification algorithm provides 97.89% of accuracy. However, this outcome is better than the recent work. The SMOTE is utilized for the balancing of the dataset.

show abstract

Performance Evaluation of Merging Techniques for Handling Small Size Files in HDFS

Cited by 4 publications

References 14 publications

A Dynamic Repository Approach for Small File Management With Fast Access Time on Hadoop Cluster: Hash Based Extended Hadoop Archive

A Dynamic Repository Approach for Small File Management With Fast Access Time on Hadoop Cluster: Hash Based Extended Hadoop Archive

Ocular Disease Severity Identification and Performance Optimisation using Custom Net Model

A Hybrid Model: Random Classification and Feature Selection Approach for Diagnosis of the Parkinson Syndrome

Contact Info

Product

Resources

About