Efficient Range-based Query Processing on the Hadoop Distributed File System

Kim, Jong Wook; Cho, Sae-Hong; Kim, Il-Min

doi:10.14257/astl.2013.43.23

Cited by 2 publications

(2 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…SkewTune is a partition algorithm proposed by Kwon et al [17], which can remit the skew of intermediate data by redistricting the larger partition of output data from map tasks. This method is similar to the improvement of the Range algorithm [25], which is also used as a comparable algorithm in our experiments. SkewTune cannot pinpoint or split exact large keys because it does not sample any key frequency information.…”

Section: Related Workmentioning

confidence: 99%

“…In this method, the intermediate < key, value > tuples are first sorted by key, and then the tuples are sequentially assigned to the reduce task according to this key range. Because the partitions may be split or combined in to different reduce tasks, this algorithm can ensure the furthest load balance of the reduce tasks [6], [25]. SARS (Self-Adaptive Reduce Scheduling).…”

Section: Experiments Settingmentioning

confidence: 99%

See 1 more Smart Citation

A Data Skew Oriented Reduce Placement Algorithm Based on Sampling

2020

IEEE Trans. Cloud Comput.

View full text Add to dashboard Cite

For frequent disk I/O and large data transmissions among different racks and physical nodes, intermediate data communication has become the most important performance bottleneck in most running Hadoop systems. This paper proposes a reduce placement algorithm called CORP to schedule related map and reduce tasks on the near nodes of clusters or racks for data locality. Because the number of keys cannot be counted until the input data are processed by map tasks, this paper applies a reservoir algorithm for sampling the input data, which can bring the distribution of keys/values closer to the overall situation of original data. Based on the distribution matrix of the intermediate results in each partition, by calculating the distance and cost matrices among the cross node communication, the related map and reduce tasks can be scheduled to relatively nearby physical nodes for data locality. We implement CORP in Hadoop 2.4.0 and evaluate its performance using three widely used benchmarks: Sort, Grep, and Join. In these experiments, an evaluation model is proposed for selecting the appropriate sample rates, which can comprehensively consider the importance of cost, effect, and variance in sampling. Experimental results show that CORP can not only improve the balance of reduces tasks effectively but also decreases the job execution time for the lower inner data communication. Compared with some other reduce scheduling algorithms, the average data transmission of the entire system on the core switch has been reduced substantially.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Experiments Settingmentioning

confidence: 99%