Algorithm and system co-design for efficient subgraph-based graph representation learning

Abstract

Subgraph-based graph representation learning (SGRL) has been recently proposed to deal with some fundamental challenges encountered by canonical graph neural networks (GNNs), and has demonstrated advantages in many important data science applications such as link, relation and motif prediction. However, current SGRL approaches suffer from scalability issues since they require extracting subgraphs for each training or test query. Recent solutions that scale up canonical GNNs may not apply to SGRL. Here, we propose a novel framework SUREL for scalable SGRL by co-designing the learning algorithm and its system support. SUREL adopts walk-based decomposition of subgraphs and reuses the walks to form subgraphs, which substantially reduces the redundancy of subgraph extraction and supports parallel computation. Experiments over six homogeneous, heterogeneous and higher-order graphs with millions of nodes and edges demonstrate the effectiveness and scalability of SUREL. In particular, compared to SGRL baselines, SUREL achieves 10X speed-up with comparable or even better prediction performance; while compared to canonical GNNs, SUREL achieves 50% prediction accuracy improvement.

SUREL+: Moving from Walks to Sets for Scalable Subgraph-based Graph Representation Learning

“…r f 6 8 j p p X 1 W 9 6 2 q t W a v U S R 5 H E c 7 g H C 7 B g x u o w z 0 0 o A U M E J 7 h F d 6 c R + f F e X c + l q 0 F J 5 8 5 h T 9 w P n 8 A 2 m G M 4 A = = < / l a t e x i t > u < l a t e x i t s h a 1 _ b a s e 6 4 = " 9 / b v X 4 5 X T 9 a Z V + w 8 r s Y i k q J X / L s = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 1 G P B i 8 c W 7 A e 0 o W y 2 k 3 b t Z h N 2 N 4 U S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 ple for Link Prediction. SUREL+ samples node sets while SEAL [54,57] extracts the whole subgraph for each query and SUREL [51] samples walks. To serve node set-based representations, SUREL+ designs a new algorithm with dedicated system support.…”

“…Given a set of nodes of interest, namely a queried node-set, SGRL models such as SEAL [54,57], GraIL [41], and SubGNN [1] first extract a subgraph around the queried node-set (termed query-induced subgraph), and then encode the extracted subgraph for prediction. Extensive works have shown that SGRL models are more robust [52] and more expressive [5,12]; while canonical graph neural networks (GNNs) including GCN [23] and GraphSAGE [14] generally fail to make accurate predictions, due to their limited expressive power [9,13,57], incapability of capturing intra-node distance information [28,39], and improper entanglement between receptive field size and model depth [18,51,52]. An example in Fig.…”

“…SUREL [51] is the state-of-the-art (SOTA) framework that applies algorithm and system co-design to implement SGRL. It employs z < l a t e x i t s h a 1 _ b a s e 6 4 = " Q 6 j 6 O 1 G k W D 7 g / 1 G K O y Y J T F s f F Y A = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m k q M e C F 4 8 t 2 F Z o Q 9 l s J + 3 a z S b s b s Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g p r 6 x u b W 8 X t 0 s 7 u 3 v 5 B + f C o r e N U M W y x W M T q P q A a B Z f Y M t w I v E 8 U 0 i g Q 2 A n G N z O / 8 4 h K 8 1 j e m U m C f k S H k o e c U W O l 5 l O / X H G r 7 h x k l X g 5 q U C O R r / 8 1 R v E L I 1 Q G i a o 1 l 3 P T Y y f U W U 4 E z g t 9 V K N C W V j O s S u p Z J G q P 1 s f u i U n F l l Q M J Y 2 Z K G z N X f E x m N t J 5 E g e 2 M q B n p Z W 8 m / u d 1 U x N e + x m X S W p Q s s W i M B X E x G T 2 N R l w h c y I i S W U K W 5 v J W x E F W X G Z l O y I X j L L 6 + S 9 k X V u 6 z W m r V K n e R x F O E E T u E c P L i C O t x C A 1 r A A O E Z X u H N e X B e n H f n Y 9 F a c P K Z Y / g D 5 / M H 3 u 2 M 4 w = = < / l a t e x i t > x < l a t e x i t s h a 1 _ b a s e 6 4 = " Q 6 j 6 O 1 G k W D 7 g / 1 G K O y Y J T F s f F Y A = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m k q M e C F 4 8 t 2 F Z o Q 9 l s J + 3 a z S b s b s Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g p r 6 x u b W 8 X t 0 s 7 u 3 v 5 B + f C o r e N U M W y x W M T q P q A a B Z f Y M t w I v E 8 U 0 i g Q 2 A n G N z O / 8 4 h K 8 1 j e m U m C f k S H k o e c U W O l 5 l O / X H G r 7 h x k l X g 5 q U C O R r / 8 1 R v E L I 1 Q G i a o 1 l 3 P T Y y f U W U 4 E z g t 9 V K N C W V j O s S u p Z J G q P 1 s f u i U n F l l Q M J Y 2 Z K G z N X f E x m N t J 5 E g e 2 M q B n p Z W 8 m / u d 1 U x N e + x m X S W p Q s s W i M B X E x G T 2 N R l w h c y I i S W U K W 5 v J W x E F W X G Z l O y I X j L L 6 + S 9 k X V u 6 z W m r V K n e R x F O E E T u E c P L i C O t x C A 1 r A A O E Z X u H N e X B e n H f n Y 9 F a c P K Z Y / g D 5 / M H 3 u 2 M 4 w = = < / l a t e x i t > x < l a t e x i t s h a 1 _ b a s e 6 4 = " L 8 3 j e x n T / i x c 5 F i L R 8 n V H l G L A C 8 = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m k q M e C F 4 8 t 2 A 9 o Q 9 l s J + 3 a z S b s b o R Q + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e F R + f i k r e N U M W y x W M S q G 1 C N g k t s G W 4 E d h O F N A o E d o L J 3 d z v P K H S P J Y P J k v Q j + h I 8 p A z a q z U z A b l i l t 1 F y D r x M t J B X I 0 B u W v / j B m a Y T S M E G 1 7 n l u Y v w p V Y Y z g b N S P 9 W Y U D a h I + x Z K m m E 2 p 8 u D p 2 R C 6 s M S R g r W 9 K Q h f p 7 Y k o j r b M o s J 0 R N W O 9 6 s 3 F / 7 x e a s J b f 8 p l k h q U b L k o T A U x M Z l / T Y Z c I T M i s 4 Q y x e 2 t h I 2 p o s z Y b E o 2 B G / 1 5 X X S v q p 6 1 9 V a s 1 a p k z y O I p z B O V y C B z d Q h 3 t o Q A s Y I D z D K 7 w 5 j 8 6 L 8 + 5 8 L F s L T j 5 z C n / g f P 4 A 4 H G M 5 A = = < / l a t e x i t > y < l a t e x i t s h a 1 _ b a s e 6 4 = " L 8 3 j e x n T / i x c 5 F i L R 8 n V H l G L A C 8 = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m k q M e C F 4 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m k q M e C F 4 8 t 2 F Z o Q 9 l s J + 3 a z S b s b o Q a + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g p r 6 x u b W 8 X t 0 s 7 u 3 v 5 B + f C o r e N U M W y x W M T q P q A a B Z f Y M t w I v E 8 U 0 i g Q 2 A n G N z O / 8 4 h K 8 1 j e m U m C f k S H k o e c U W O l 5 l O / X H G r 7 h x k l X g 5 q U C O R r / 8 1 R v E L I 1 Q G i a o 1 l 3 P T Y y f U W U 4 E z g t 9 V K N C W V j O s S u p Z J G q P 1 s f u i U n F l l Q M J Y 2 Z K G z N X f E x m N t J 5 E g...…”

DyHNet: Learning Dynamic Heterogeneous Network Representations

“…To improve SubGNN by distinguishing nodes inside and outside the subgraph, GNN with LAbeling trickS for Subgraph (GLASS) [37] utilizes an expressive and scalable labeling trick to enhance GNNs for subgraph representation learning. Most recently, to address the scalability issue in the subgraph representation learning problem via GNNs, SUREL [38] reduces the redundancy of subgraph extraction and supports parallel processing by decoupling the graph structure into sets of walks and reusing the walks to form subgraphs.…”

Learning Heterogeneous Subgraph Representations for Team Discovery

“…To improve SubGNN by distinguishing nodes inside and outside the subgraph, GLASS [50], GNN with LAbeling trickS for Subgraph, utilizes an expressive and scalable labelling trick to enhance GNNs for subgraph representation learning. Very recently, to address the scalability issue in the subgraph representation learning problem via GNNs, SUREL [51] reduces the redundancy of subgraph extraction and supports parallel processing by decoupling the graph structure into sets of walks and reusing the walks to form subgraphs.…”