An Algorithm of Data Skew in Spark Based on Partition

Shi, Xiaohu; Yueqin, Qian

doi:10.1109/cipae51077.2020.00063

Cited by 1 publication

(1 citation statement)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the proposed algorithm was more suitable for the case where the computing power of each node was the same. Xiujin and Yueqin [35] improved the aforementioned method by adding a dynamic evaluation method for the computing resource capability to more accurately evaluate the state of each computing node and more scientifically allocate the data stream to partitions with different resource allocations. Wang and Khan [36] predicted automatically the potential problems a priori based on limited execution data and recommended the use of a locality setting and partitions.…”

Section: Introductionmentioning

confidence: 99%

A Cluster-Based Partition Method of Remote Sensing Data for Efficient Distributed Image Processing

Chen

et al. 2022

Remote Sensing

View full text Add to dashboard Cite

Data stream partitioning is a fundamental and important mechanism for distributed systems. However, use of an inappropriate partition scheme may generate a data skew problem, which can influence the execution efficiency of many application tasks. Processing of skewed partitions usually takes a longer time, need more computational resources to complete the task and can even become a performance bottleneck. To solve such data skew issues, this paper proposes a novel partition method to divide on demand the image tiles uniformly into partitions. The partitioning problem is then transformed into a uniform and compact clustering problem whereby the image tiles are regarded as image pixels without spectrum and texture information. First, the equal area conversion principle was used to select the seed points of the partitions and then the image tiles were aggregated in an image layout, thus achieving an initial partition scheme. Second, the image tiles of the initial partition were finely adjusted in the vertical and horizontal directions in separate steps to achieve a uniform distribution among the partitions. Two traditional partition methods were adopted to evaluate the efficiency of the proposed method in terms of the image segmentation testing, data shuffle testing, and image clipping testing. The results demonstrated that the proposed partition method solved the data skew problem observed in the hash partition method. In addition, this method is designed specifically for processing of image tiles and makes the related processing operations for large-scale images faster and more efficient.

show abstract

Section: Introductionmentioning

confidence: 99%

A Cluster-Based Partition Method of Remote Sensing Data for Efficient Distributed Image Processing

Chen

et al. 2022

Remote Sensing

View full text Add to dashboard Cite

show abstract

An Algorithm of Data Skew in Spark Based on Partition

Cited by 1 publication

References 16 publications

A Cluster-Based Partition Method of Remote Sensing Data for Efficient Distributed Image Processing

A Cluster-Based Partition Method of Remote Sensing Data for Efficient Distributed Image Processing

Contact Info

Product

Resources

About