How Expressive are Transformers in Spectral Domain for Graphs?

Bastos, Anson; Nadgeri, Abhishek; Singh, Kuldeep; Kanezashi, Masakoto; Suzumura, Toyotaro; Mulang', Isaiah Onando

doi:10.48550/arxiv.2201.09332

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Graph Transformer is inspired by the Transformer architecture, which has shown remarkable performance in natural language processing [65,13,41]. The Graph Transformer extends the Transformer architecture to the graph domain, allowing the model to capture the global structure and long-range dependencies of the graph [86,15,31,29,74,48,42,80,8,43,11,5,23,88].…”

Section: B3 Detalied Related Workmentioning

confidence: 99%

GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning

Zhao

Liu²,

et al. 2023

Preprint

View full text Add to dashboard Cite

Molecule property prediction has gained significant attention in recent years. The main bottleneck is the label insufficiency caused by expensive lab experiments. In order to alleviate this issue and to better leverage textual knowledge for tasks, this study investigates the feasibility of employing natural language instructions to accomplish molecule-related tasks in a zero-shot setting. We discover that existing molecule-text models perform poorly in this setting due to inadequate treatment of instructions and limited capacity for graphs. To overcome these issues, we propose GIMLET, which unifies language models for both graph and text data. By adopting generalized position embedding, our model is extended to encode both graph structures and instruction text without additional graph encoding modules. GIMLET also decouples encoding of the graph from tasks instructions in the attention mechanism, enhancing the generalization of graph features across novel tasks. We construct a dataset consisting of more than two thousand molecule tasks with corresponding instructions derived from task descriptions. We pretrain GIMLET on the molecule tasks along with instructions, enabling the model to transfer effectively to a broad range of tasks. Experimental results demonstrate that GIMLET significantly outperforms molecule-text baselines in instruction-based zero-shot learning, even achieving closed results to supervised GNN models on tasks such as toxcast and muv.

show abstract

Section: B3 Detalied Related Workmentioning

confidence: 99%

GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning

Zhao

Liu²,

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Some other works introduce structure information into attention by graph distance, path embedding or feature encoded by GNN (Park et al, 2022;Maziarka et al, 2020;Ying et al, 2021;Mialon et al, 2021;Choromanski et al, 2022). Other works use transformer as a module of the whole model (Bastos et al, 2022;Guo et al, 2022).…”

Section: Related Workmentioning

confidence: 99%

Assessing Digital Literacy

Zhang¹

2021

Peking University Linguistics Research

View full text Add to dashboard Cite

Research (PKULR) is a cooperation project between Springer Nature and Peking University Press. This series presents the latest discoveries and developments of significance in linguistic research conducted by famous Chinese scholars. Titles in this series are carefully evaluated, examined and selected by Peking University (which ranks No. 10 in the world and No. 1 in China in the QS World University Rankings-Linguistics 2016) and Peking University Press (which was honored as the most influential publisher in linguistics according to Chinese Book Citation Index, 2016), covering all major aspects of linguisticsphonetics, phonology, pragmatics, semantics, morphology, syntax, theoretical linguistics, applied linguistics and inter-disciplinary studies. PKULR aims to provide an invaluable guide to the very nature of language. On the one hand, it tries to offer a thorough grounding in the fundamental concepts of linguistics; on the other hand, it also attaches great importance to the practical application of these concepts, esp. in Chinese context.

show abstract

How Expressive are Transformers in Spectral Domain for Graphs?

Cited by 2 publications

References 0 publications

GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning

GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning

Assessing Digital Literacy

Contact Info

Product

Resources

About