PaSca: A Graph Neural Architecture Search System under the Scalable Paradigm

Zhang, Wentao; Shen, Yu; Lin, Zheyu; Li, Yang; Li, Xiao-Sen; Ouyang, Wen; Tao, Yangyu; Yang, Zhi; Cui, Bin

doi:10.1145/3485447.3511986

Cited by 41 publications

(20 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The resulted DAG has 9 choices, as illustrated in Fig 1 . Those intermediate nodes without successor nodes are connected to the output node by concatenation. Besides this macro space, we also consider optional fully-connected pre-process and post-process layers as in [60,62]. Notice that to avoid exploding the search space, we consider the numbers of pre-process and post-process layers as hyper-parameters, which will be discussed in Section 3.3.…”

Section: Search Space Designmentioning

confidence: 99%

NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search

Qin¹,

Zhang²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Graph neural architecture search (GraphNAS) has recently aroused considerable attention in both academia and industry. However, two key challenges seriously hinder the further research of GraphNAS. First, since there is no consensus for the experimental setting, the empirical results in different research papers are often not comparable and even not reproducible, leading to unfair comparisons. Secondly, GraphNAS often needs extensive computations, which makes it highly inefficient and inaccessible to researchers without access to large-scale computation. To solve these challenges, we propose NAS-Bench-Graph, a tailored benchmark that supports unified, reproducible, and efficient evaluations for GraphNAS. Specifically, we construct a unified, expressive yet compact search space, covering 26,206 unique graph neural network (GNN) architectures and propose a principled evaluation protocol. To avoid unnecessary repetitive training, we have trained and evaluated all of these architectures on nine representative graph datasets, recording detailed metrics including train, validation, and test performance in each epoch, the latency, the number of parameters, etc. Based on our proposed benchmark, the performance of GNN architectures can be directly obtained by a look-up table without any further computation, which enables fair, fully reproducible, and efficient comparisons. To demonstrate its usage, we make in-depth analyses of our proposed NAS-Bench-Graph, revealing several interesting findings for GraphNAS. We also showcase how the benchmark can be easily compatible with GraphNAS open libraries such as AutoGL and NNI. To the best of our knowledge, our work is the first benchmark for graph neural architecture search.

show abstract

Section: Search Space Designmentioning

confidence: 99%

NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search

Qin¹,

Zhang²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…PaSca [132] is a novel paradigm and system that offers Bayesian optimization to systematically construct and explore the design space for scalable GNNs, rather than individual designs. PaSca proposes a novel abstraction called SGAP to address data and model scalability issues.…”

Section: Systems On Gpu Clustersmentioning

confidence: 99%

Distributed Graph Neural Network Training: A Survey

Shao¹,

Li²,

Gu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Graph neural networks (GNNs) are a type of deep learning models that learning over graphs, and have been successfully applied in many domains. Despite the effectiveness of GNNs, it is still challenging for GNNs to efficiently scale to large graphs. As a remedy, distributed computing becomes a promising solution of training large-scale GNNs, since it is able to provide abundant computing resources. However, the dependency of graph structure increases the difficulty of achieving high-efficiency distributed GNN training, which suffers from the massive communication and workload imbalance. In recent years, many efforts have been made on distributed GNN training, and an array of training algorithms and systems have been proposed. Yet, there is a lack of systematic review on the optimization techniques from graph processing to distributed execution. In this survey, we analyze three major challenges in distributed GNN training that are massive feature communication, the loss of model accuracy and workload imbalance. Then we introduce a new taxonomy for the optimization techniques in distributed GNN training that address the above challenges. The new taxonomy classifies existing techniques into four categories that are GNN data partition, GNN batch generation, GNN execution model, and GNN communication protocol. We carefully discuss the techniques in each category. In the end, we summarize existing distributed GNN systems for multi-GPUs, GPU-clusters and CPU-clusters, respectively, and give a discussion about the future direction on scalable GNNs.

show abstract

“…Most existing GNNs need to repeatedly perform the computationally expensive and recursive feature smoothing, which involves the participation of the entire graph at each training epoch. (Zhang et al, 2022) Furthermore, most methods adopt the same training loss function as GAE, which introduces high memory usage by storing the dense-form adjacency matrix on GPU. For a graph of size 200 million, its dense-form adjacency matrix requires a space of roughly 150GB, exceeding the memory capacity of the current powerful GPU devices.…”

Section: Introductionmentioning

confidence: 99%

NAFS: A Simple yet Tough-to-beat Baseline for Graph Representation Learning

Zhang¹,

Sheng²,

Yang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Recently, graph neural networks (GNNs) have shown prominent performance in graph representation learning by leveraging knowledge from both graph structure and node features. However, most of them have two major limitations. First, GNNs can learn higher-order structural information by stacking more layers but can not deal with large depth due to the over-smoothing issue. Second, it is not easy to apply these methods on large graphs due to the expensive computation cost and high memory usage. In this paper, we present node-adaptive feature smoothing (NAFS), a simple non-parametric method that constructs node representations without parameter learning. NAFS first extracts the features of each node with its neighbors of different hops by feature smoothing, and then adaptively combines the smoothed features. Besides, the constructed node representation can further be enhanced by the ensemble of smoothed features extracted via different smoothing strategies. We conduct experiments on four benchmark datasets on two different application scenarios: node clustering and link prediction. Remarkably, NAFS with feature ensemble outperforms the state-of-the-art GNNs on these tasks and mitigates the aforementioned two limitations of most learning-based GNN counterparts.

show abstract

PaSca: A Graph Neural Architecture Search System under the Scalable Paradigm

Cited by 41 publications

References 40 publications

NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search

NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search

Distributed Graph Neural Network Training: A Survey

NAFS: A Simple yet Tough-to-beat Baseline for Graph Representation Learning

Contact Info

Product

Resources

About