MapReduce has become an effective approach to big data analytics in large cluster systems, where SQL-like queries play important roles to interface between users and systems. However, based on our Facebook daily operation results, certain types of queries are executed at an unacceptable low speed by Hive (a production SQL-to-MapReduce translator). In this paper, we demonstrate that existing SQL-to-MapReduce translators that operate in a one-operation-to-one-job mode and do not consider query correlations cannot generate high-performance MapReduce programs for certain queries, due to the mismatch between complex SQL structures and simple MapReduce framework. We propose and develop a system called YSmart, a correlation aware SQL-to-MapReduce translator. YSmart applies a set of rules to use the minimal number of MapReduce jobs to execute multiple correlated operations in a complex query. YSmart can significantly reduce redundant computations, I/O operations and network transfers compared to existing translators. We have implemented YSmart with intensive evaluation for complex queries on two Amazon EC2 clusters and one Facebook production cluster. The results show that YSmart can outperform Hive and Pig, two widely used SQL-to-MapReduce translators, by more than four times for query execution.

No abstract

No abstract

Based on the statistical property of SAR image speckle noise and the property that the multiscale geometric analysis can capture the intrinsic geometrical structure of image, combining curvelet transform with BivaShrink denoising model, a method of SAR image denoising based on curvelet domain is presented in this paper. According to calculation of variance homogeneous measurement and curvelet coefficients of current layer and its parent layer, the local adaptive window is determined to optimally estimate shrinkage factor. The method can effectively reduce SAR speckle noise and preserving details of SAR image well through the correlation of curvelet coefficients in the same direction of subband and parent-layer subband. The experimental results show the presented method greatly improves the subjective visual effect and the numerical indicators of the denoised image.