Large-Scale Graph Processing Using Apache Giraph

Sakr, Sherif; Orakzai, Faisal Moeen; Abdelaziz, Ibrahim; Khayyat, Zuhair

doi:10.1007/978-3-319-47431-1

Cited by 23 publications

(16 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our idea is to first select some landmarks and compute their shortest path trees in the offline stage. Then, when a query of vertex v is input, we employ distributed graph systems, such as Pregel [16], Pregel+ [25], Giraph [21], and GraphX [9], to perform the distance computation between v and any other vertex based on the shortest path trees of the selected landmarks.…”

Section: A Landmark-based Distance Computation Framework Over DImentioning

confidence: 99%

“…As the sizes of real graphs increase, the classic solutions for SSSP length queries in real graphs become inefficient. Additionally, the emergence of distributed graph systems, including Pregel [16], Pregel+ [25], Giraph [21] and GraphX [9], is inevitable to maintain large graphs. These distributed graph systems follow the vertex-centric BSP (bulk synchronous parallel) computing model, which divides the calculation into a series of superstep iterations.…”

Section: Introductionmentioning

confidence: 99%

“…Although we prove that the problem of selecting the optimal set of landmarks is NP-hard, we propose a heuristic distributed strategy to guarantee the approximation ratio. • Finally, we conduct extensive experiments over different kinds of real graphs on multiple distributed graph systems (Pregel+ [25], Giraph [21] and GraphX [9]) to verify the performance of the proposed techniques.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Optimizing Distance Computation in Distributed Graph Systems

Wang

Peng

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Given a large graph, such as a social network or a knowledge graph, a fundamental query is how to find the distance from a source vertex to another vertex in the graph. As real graphs become very large and many distributed graph systems, such as Pregel [16], Pregel+ [25], Giraph [21], and GraphX [9], are proposed, how to employ distributed graph systems to process single-source distance queries should attract more attention. In this paper, we propose a landmark-based framework to optimize the distance computation over distributed graph systems. We also use a measure called set betweenness to select the optimal set of landmarks for distance computation. Although we can prove that selecting the optimal set of landmarks is NP-hard, we propose a heuristic distributed algorithm that can guarantee the approximation ratio. Experiments on large real graphs confirm the superiority of our methods.

show abstract

Section: A Landmark-based Distance Computation Framework Over DImentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Optimizing Distance Computation in Distributed Graph Systems

Wang

Peng

et al. 2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…In particular, for about a decade, the Hadoop platform represented the defacto standard of Big Data analytics world. Though, recently, we have been witnessing a new wave of Big Data 2.0 processing platforms [38] that are dedicated to specific verticals such as structured SQL data processing (e.g., Hive [44], Impala [23], Presto 1 ), large scale graph processing (e.g., Giraph [39], Graphlab [30], GraphX [17] and large scale stream processing data (e.g., Storm 2 , Heron [26], Flink [12], Samza [35], Kafka [25]).…”

Section: Big Data Sciencementioning

confidence: 99%

Big Data Systems Meet Machine Learning Challenges: Towards Big Data Science as a Service

et al. 2018

Self Cite

View full text Add to dashboard Cite

Recently, we have been witnessing huge advancements in the scale of data we routinely generate and collect in pretty much everything we do, as well as our ability to exploit modern technologies to process, analyze and understand this data. The intersection of these trends is what is called, nowadays, as Big Data Science. Cloud computing represents a practical and cost-effective solution for supporting Big Data storage, processing and for sophisticated analytics applications. We analyze in details the building blocks of the software stack for supporting big data science as a commodity service for data scientists. We provide various insights about the latest ongoing developments and open challenges in this domain. 1 https://prestodb.io/ 2

show abstract

“…To overcome these limitations, it is necessary to resort to distributed graph process. While some distributed graph process frameworks (e.g., MapReduce [17], Pregel [18] and Apache Giraph [19])…”

Section: B Introduction On Graph Analysis and Embedding Algorithmsmentioning

confidence: 99%

Cognitive satellite communications and representation learning for streaming and complex graphs.

Liu¹

View full text Add to dashboard Cite

This dissertation includes two separate topics. The first topic studies a promising dynamic spectrum access algorithm that improves the throughput of satellite communication (SATCOM) under the uncertainty environment. The other topic investigates real-time distributed representation learning for streaming and complex networks. 1 Cognitive Satellite Communications Dynamic spectrum access (DSA) allows a secondary user to access the spectrum holes that are not occupied by primary users. However, DSA is normally operated under uncertainty in a complex SATCOM environment, which could cause more spectrum sensing errors or even service disruption. In this case, DSA requires a decision-making process to optimally determine which channels to sense and access. To this end, I propose a solution that addresses the uncertainty in SATCOM to maximize the system throughput. Specifically, the DSA decision making process is formulated as a Partially Observable Markov Decision Process (POMDP) model. Simulation results prove the effectiveness of our proposed DSA strategy. v 2 Distributed Real-time Representation Learning of Large Networks Large-scale networks have attracted significant amount of attentions to extract and analyze the hidden information from big data. In particular, graph embedding learns the representations of the original network in a lower vector space while maximally preserving the original structural information and the similarity among nodes. I propose a real-time distributed graph embedding algorithm (RTDGE) which is capable of distributively embedding the streaming graph data by combining a novel edge partition approach and an incremental negative sample approach. Furthermore, a real-time distributed streaming data processing platform is prototyped based on Kafka and Storm. On this platform, real-time Twitter network data can be retrieved, partitioned and processed for state-of-art tasks including synonymic user detection, community classification and visualization. For complex knowledge graphs, existing works cannot capture the complex connection patterns and never consider the impacts from complicated relations, due to the unquantifiable relationships. In this dissertation, a novel hierarchical embedding algorithm is proposed to hierarchically measure the structural similarities and the impacts from relations by constructing a multi-layer graph. Then an advanced representation learning model is designed based on an entity's context, which is generated by taking random walks on the multi-layer content graph. Experimental results show that our proposed model outperforms the state-of-the-art techniques. vi

show abstract

Large-Scale Graph Processing Using Apache Giraph

Cited by 23 publications

References 24 publications

Optimizing Distance Computation in Distributed Graph Systems

Optimizing Distance Computation in Distributed Graph Systems

Big Data Systems Meet Machine Learning Challenges: Towards Big Data Science as a Service

Cognitive satellite communications and representation learning for streaming and complex graphs.

Contact Info

Product

Resources

About