Implementing WebGIS on Hadoop: A case study of improving small file I/O performance on HDFS

Liu, Xuhui; Han, Jizhong; Zhong, Yue; Han, Chengde; He, Xubin

doi:10.1109/clustr.2009.5289196

Cited by 149 publications

(74 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Reference [9] and [10] were put forward their own solutions in view of the different application, in fact the thought of the two papers is similar: the basic idea is to combine small files into large ones to reduce the file number and build index for each file. in the original HDFS basis to add a small document processing module, when a file arrived, judge whether the file belongs to a small file, if it is, it gave it to the document processing module processing, otherwise store in HDSF directly.…”

Section: Other Solutionsmentioning

confidence: 99%

“…Papers [9] and [10] which mentioned in Section four are proposed for specific applications solutions, they are not common technical solutions.…”

Section: Conclusion and Proposed Solutionmentioning

confidence: 99%

“…Paper [10] introduces a novel approach to improve the efficiency of storing and accessing small files on HDFS. Characteristics of file correlations and access locality remaining among small files in the context of courseware are well considered for storing and accessing them.…”

Section: Other Solutionsmentioning

confidence: 99%

See 2 more Smart Citations

Research on the Small Files Problem of Hadoop

Liu¹,

Peng²,

Yu³

2015

Advances in Social Science, Education and Humanities Research

View full text Add to dashboard Cite

Abstract. Although Hadoop is widely used, its full potential is not yet put to use because of some issues, the small files problem being one of them. Firstly, the paper analyses the causes of the small files problem of Hadoop. Then, the current program to solve the small files problem are introduced, including Hadoop own programs and other application-specific solutions, and analyzes the advantages and disadvantages of various options. Finally, we present two research ideas, one is to use a combination of RDBMS and Hadoop; Another is to make the "Datanode" caching some metadata of the small files.

show abstract

Section: Other Solutionsmentioning

confidence: 99%

“…Papers [9] and [10] which mentioned in Section four are proposed for specific applications solutions, they are not common technical solutions.…”

Section: Conclusion and Proposed Solutionmentioning

confidence: 99%

See 1 more Smart Citation

Research on the Small Files Problem of Hadoop

Liu¹,

Peng²,

Yu³

2015

Advances in Social Science, Education and Humanities Research

View full text Add to dashboard Cite

show abstract

“…Preprocessing is also required for each addition of new input image set. Small files cannot be directly accessed in SequenceFile, whole SequenceFile has to be processed to obtain an image data as one element [14].…”

Section: L a S Smentioning

confidence: 99%

Hadoop Optimization for Massive Image Processing: Case Study Face Detection

Demir

Sayar

2014

INT J COMPUT COMMUN

View full text Add to dashboard Cite

Abstract:Face detection applications are widely used for searching, tagging and classifying people inside very large image databases. This type of applications requires processing of relatively small sized and large number of images. On the other hand, Hadoop Distributed File System (HDFS) is originally designed for storing and processing largesize files. Huge number of small-size images causes slowdown in HDFS by increasing total initialization time of jobs, scheduling overhead of tasks and memory usage of the file system manager (Namenode). The study in this paper presents two approaches to improve small image file processing performance of HDFS. These are (1) converting the images into single large-size file by merging and (2) combining many images for a single task without merging. We also introduce novel Hadoop file formats and record generation methods (for reading image content) in order to develop these techniques.

show abstract

“…Even the improved LATE speculative execution could end up increasing the degree of congestion within a busy cluster, if speculative copies are simply assigned to machines that are already close to maximum resource utilization. Resource Aware Programming in Hadoop has prettify one of the Explore Challenges [14] [15] in Cloud Computing. Programming in Hadoop is centralized, and initiated.…”

Section: Resource Aware Schedulingmentioning

confidence: 99%

Recent Job Scheduling Algorithms in Hadoop Cluster Environments: A Survey

Patil¹,

Bagban²,

Pande³

2015

International Journal of Advanced Research in Computer and Comm

View full text Add to dashboard Cite

This paper discusses Cloud Computing is rising as a replacement machine paradigm shift. HadoopMapReduce has become a robust Computation Model for process giant knowledge on distributed commodity hardware clusters like Clouds. Studies describe that Hadoop implementations, the default first in first out scheduler hardware is accessible wherever jobs are scheduled in FIFO order with support for different priority primarily based schedulers also other defaulter schedulers . During this paper we have studied numerous scheduling enhancements in a scheduling techniques a brand new with Hadoop like Fair4s scheduling algorithm with its extended functionalities allows processing large as well small jobs with effective fairness without starvation of small jobs. Majority of small jobs are available till date so Job scheduling algorithm must take in consideration of small jobs first with effective major to process these jobs.

show abstract

Implementing WebGIS on Hadoop: A case study of improving small file I/O performance on HDFS

Cited by 149 publications

References 8 publications

Research on the Small Files Problem of Hadoop

Research on the Small Files Problem of Hadoop

Hadoop Optimization for Massive Image Processing: Case Study Face Detection

Recent Job Scheduling Algorithms in Hadoop Cluster Environments: A Survey

Contact Info

Product

Resources

About