SpeCH: A scalable framework for data placement of data-intensive services in geo-distributed clouds

Abstract

The advent of big data analytics and cloud computing technologies has resulted in wide-spread research on the data placement problem. Since data-intensive services require access to multiple datasets within each transaction, traditional schemes of uniformly partitioning the data into distributed nodes, as employed by many popular data stores like HDFS or Cassandra, may cause network congestion thereby affecting system throughput. In this article, we propose a scalable and unified framework for data-intensive service data placement into geographically distributed clouds. The proposed framework introduces a new paradigm for partitioning a set of data-items into geo-distributed clouds using Spectral Clustering on Hypergraphs, and is therefore called SpeCH.Scaling spectral methods to large workloads is challenging, since computing the spectra of the hypergraph laplacian is a computationally intensive task. SpeCH provides two solutions to tackle this problem: (1) an algorithm, called SpectralApprox, that leverages randomized techniques for obtaining low-rank approximations of the hypergraph matrix with bounded guarantees, thereby significantly improving the efficiency of spectral clustering while also providing high quality solutions in practice; (2) an algorithm, called SpectralDist, that exploits the highly parallel nature of the spectral clustering algorithm and uses Apache Spark to speed-up the process while retaining the same quality guarantees as the exact algorithm. Additionally, being distributed in nature, Spec-tralDist enables SpeCH to perform data placement on workloads that require resources beyond the capacity of a single machine. Experiments on a real-world trace-based online social network dataset show that the SpeCH is effective, efficient, and scalable. Empirically, SpectralApprox is comparable in efficacy on the evaluated metrics, while being up to 10 times faster in execution time when compared to state-of-the-art techniques. On the other hand, though SpectralApprox is 7-8 times faster when compared to SpectralDist, in terms of efficacy on the evaluated metrics the latter is up to 50% better.

UnifyDR: A Generic Framework for Unifying Data and Replica Placement

“…More specifically, any change (small or large) in the system workload would require re-execution of the full pipeline to obtain the placement output. This design decision is in line with almost every existent technique [3]- [5], [16], [17], [43], [49] in the extensive literature on data placement. Thus, making the CDR placement algorithm dynamically adapt to the changes in the system workload is not in the scope of the current work.…”

“…On the other hand, publicly available specialized heuristics for hypergraph partitioning [7] enable graceful scaling of the aforementioned methods to large datasets. Moving further, Atrey et al [3], [5] proposed an algorithm based on spectral clustering of hypergraphs, which portrayed quality similar to the algorithms proposed in [43], however, achieved superior efficiency and scalability owing to the use of randomized eigendecomposition techniques for factorizing the hypergraph laplacian.…”

“…Although advancements in enabling technologies such as big data and cloud computing have provided us with the necessary machinery and systems (e.g., Apache Hadoop [39] and Spark [46]) to perform data management at scale, effective strategies for data placement and partitioning remain crucial for ensuring the performance of such systems [15]. Having said that, the field of data placement has witnessed a humongous amount of research over the past two decades [3], [4], [17], [30], [36], [41], [43], [45], [48], [49].…”

“…Therefore, both data and replica placement should be considered as objectives of a single joint optimization problem. Having said that, although the field of data placement has witnessed significant advancements [3], [17], [43], to the best of our knowledge, none of the existing techniques possess the capability of performing data and replica placement jointly. On the contrary, most of the existing techniques treat the two placement steps as independent problems, and perform data placement followed by replica placement (Fig.…”

“…Owing to their ability to capture both data-item -node and data-item -data-item associations, the methods proposed by Atrey et al [3] and Yu et al [43] (Hyper)graph based solutions have also been popular for data placement of more traditional workloads such as scientific and relational workflows. The existence of a polynomialtime reduction of the data placement problem into an instance of the graph partitioning problem was proved in [17].…”