Optimize Multidimensional Arrays Queries with Heterogeneous Replica Method

Abstract

Multidimensional arrays are commonly used in scientific and engineering applications. The disk layout for the multidimensional arrays will obviously affect the performance of data querying. Homogeneous Replica method are widely used to maintain the data reliability in most of the distributed storage systems and used to improve the data locality in some parallel processing systems. In this paper, we propose a novel method, that is heterogeneous replicas, to makes better use of the replica method to optimize the performance of multidimensional arrays querying. The experimental results shows that heterogeneous replicas method can significantly reduce the overhead of disk I/O for most of the queries. With three heterogeneous replicas, the performance of random generated range queries for multidimensional datasets can be improved for 30% on the average.

Segmentation of Chinese Web Text Based on Spark

“…Spark uses the HDFS Hadoop as the file system [3], but the difference is that the design of Spark takes into account the work load of direct reuse of work data sets in parallel operation of the cluster computing. The work load is optimized by the introduction of memory cluster computing, and the data set is delayed in memory , and the access latency is shortened; Based on this idea, a set of tachyon memory file system with 100 times higher throughput than HDFS is also developed.…”