Hadoop Massive Small File Merging Technology Based on Visiting Hot-Spot and Associated File Optimization

Hadoop Perfect File: A fast and memory-efficient metadata access archive file to face small files problem in HDFS

“…Some suggested solutions include altering HDFS by adding hardware to speed up small file processing or letting HDFS automatically combine small files before storage. Peng et al proposed the Small Hadoop Distributed File System (SHDFS) [28], which is based on the original HDFS but includes a merging and caching module. The merging module employs a correlated files model to identify and merge correlated files using user-based collaborative filtering.…”