All in One: Multi-Task Prompting for Graph Neural Networks

Abstract

Recently, "pre-training and fine-tuning" has been adopted as a standard workflow for many graph tasks since it can take general graph knowledge to relieve the lack of graph annotations from each application. However, graph tasks with node level, edge level, and graph level are far diversified, making the pre-training pretext often incompatible with these multiple tasks. This gap may even cause a "negative transfer" to the specific application, leading to poor results. Inspired by the prompt learning in natural language processing (NLP), which has presented significant effectiveness in leveraging prior knowledge for various NLP tasks, we study the prompting topic for graphs with the motivation of filling the gap between pretrained models and various graph tasks. In this paper, we propose a novel multi-task prompting method for graph models. Specifically, we first unify the format of graph prompts and language prompts with the prompt token, token structure, and inserting pattern. In this way, the prompting idea from NLP can be seamlessly introduced to the graph area. Then, to further narrow the gap between various graph tasks and state-of-the-art pre-training strategies, we further study the task space of various graph applications and reformulate downstream problems to the graph-level task. Afterward, we introduce meta-learning to efficiently learn a better initialization for the multi-task prompt of graphs so that our prompting framework can be more reliable and general for different tasks. We conduct extensive experiments, results from which demonstrate the superiority of our method.

MultiGPrompt for Multi-Task Pre-Training and Prompting on Graphs

“…We present more details of the baselines in Appendix B. It is worth noting that certain few-shot methodologies on graphs, such as Meta-GNN [64], AMM-GNN [47], RALE [29], VNT [42], and ProG [41], hinge on the meta-learning paradigm [15], requiring an additional set of labeled base classes in addition to the few-shot classes. Hence, they are not comparable to our framework.…”

“…Due to the parameter-efficient nature of prompt, it has been quickly popularized in favor of finetuning larger pre-trained models, or when the downstream task only has few-shot labels. Given the advantages, prompt-based learning has also been explored on graphs [30,[40][41][42]58].…”

“…Specifically, GraphPrompt [30] and ProG [41] attempt to unify pre-training and typical downstream tasks on graphs into a common template, and further introduce learnable prompts to guide the downstream tasks. Note that ProG requires a set of base classes in a typical meta-learning setup [47,64], while GraphPrompt does not make use of base classes.…”

“…As a prompt entails much fewer parameters than the pre-trained model, it is especially amenable to few-shot settings where there are very limited task-specific labels. Following the success of prompt-based learning, researchers have also begun to explore prompt-based or related parameterefficient learning on graph data [14,24,30,[40][41][42]61]. However, most existing research in prompt-based graph learning only utilizes a single pretext task in pre-training.…”

Knowledge Graph Multi-Hop Question Answering Based on Dependent Syntactic Semantic Augmented Graph Networks

“…Simultaneously, the integration of graph neural networks (GNNs) has emerged as a pivotal innovation, enhancing the representation of complex entity relationships essential for multi-hop QA. The introduction of methods like multi-task prompting for graph neural networks highlights the potential of leveraging pre-trained models across various graph tasks, effectively bridging the gap between general graph knowledge and specific application needs [17]. The exploration of hypergraph representation for sociological analysis emphasizes the richness of social interactions and environments, providing a novel approach to understanding complex sociological phenomena through data mining techniques [18].…”