GNN-LM: Language Modeling based on Global Contexts via GNN

Meng, Yuxian; Zong, Shi; Li, Xiaoya; Sun, Xiaofei; Zhang, Tianwei; Wu, Fei; Li, Jiwei

doi:10.48550/arxiv.2110.08743

Cited by 9 publications

(9 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Recently, Graph Neural Network (GNN) has been emerged as a extremely suitable approach for processing graph-based data. As semantic graph generation from textual data has been investigated for a long time, using GNN for NLP promises a very potential direction for many related tasks [106][107][108].…”

Section: Discussionmentioning

confidence: 99%

N/A Modern Approaches in Natural Language Processing

Tho¹

2022

JCSCE

View full text Add to dashboard Cite

Natural Language Processing (NLP) is one of the major branches in the emerging field of Artificial Intelligence (AI). Classical approaches in this area were mostly based on parsing and information extraction techniques, which suffered from great difficulty when dealing with very large textual datasets available in practical applications. This issue can potentially be addressed with the recent advancement of the Deep Learning (DL) techniques, which are naturally assuming very large datasets for training. In fact, NLP research has witnessed a remarkable achievement with the introduction of Word Embedding techniques, which allows a document to be represented meaningfully as a matrix, on which major DL models like CNN or RNN can be deployed effectively to accomplish common NLP tasks. Gradually, NLP scholars keep developing specific models for their areas, notably attention-enhanced BiLSTM, Transformer and BERT. The births of those models have introduced a new wave of modern approaches which frequently report new breaking results and open much novel research directions. The aim of this paper is to give readers a roadmap of those modern approaches in NLP, including their ideas, theories and applications. This would hopefully offer a solid background for further research in this area.

show abstract

Section: Discussionmentioning

confidence: 99%

N/A Modern Approaches in Natural Language Processing

Tho¹

2022

JCSCE

View full text Add to dashboard Cite

show abstract

“…The successful implementation of AF-2 brought out possibilities for DL to resolve de novo design tasks. Currently, AI-methods such as Recurrent Neural Network (RNN) [ 116 ], CNN, Graph Neural Networks (GNN) [ 117 ] and Generative Adversarial Nets (GAN) [ 118 ] have highly participated in integrating de novo design models ( Table 4 ). Using AI-based techniques for de novo design functional proteins is showing an apparent upward trend [ 119 ].…”

Section: De Novo Design Of Food Enzymesmentioning

confidence: 99%

Possibilities of Using De Novo Design for Generating Diverse Functional Food Enzymes

Wang

Tan

et al. 2023

IJMS

View full text Add to dashboard Cite

Food enzymes have an important role in the improvement of certain food characteristics, such as texture improvement, elimination of toxins and allergens, production of carbohydrates, enhancing flavor/appearance characteristics. Recently, along with the development of artificial meats, food enzymes have been employed to achieve more diverse functions, especially in converting non-edible biomass to delicious foods. Reported food enzyme modifications for specific applications have highlighted the significance of enzyme engineering. However, using direct evolution or rational design showed inherent limitations due to the mutation rates, which made it difficult to satisfy the stability or specific activity needs for certain applications. Generating functional enzymes using de novo design, which highly assembles naturally existing enzymes, provides potential solutions for screening desired enzymes. Here, we describe the functions and applications of food enzymes to introduce the need for food enzymes engineering. To illustrate the possibilities of using de novo design for generating diverse functional proteins, we reviewed protein modelling and de novo design methods and their implementations. The future directions for adding structural data for de novo design model training, acquiring diversified training data, and investigating the relationship between enzyme–substrate binding and activity were highlighted as challenges to overcome for the de novo design of food enzymes.

show abstract

“…Retrieval-Augmented Generative Models. Using external memory to augment traditional models has recently drawn attention in natural language processing (NLP) [14,13,18,11]. For example, RETRO [4] proposes a retrieval-enhanced…”

Section: A Related Workmentioning

confidence: 99%

Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Models

Rombach¹,

Blattmann²,

Ommer³

2022

Preprint

View full text Add to dashboard Cite

Novel architectures have recently improved generative image synthesis leading to excellent visual quality in various tasks. Of particular note is the field of "AI-Art", which has seen unprecedented growth with the emergence of powerful multimodal models such as CLIP. By combining speech and image synthesis models, so-called "prompt-engineering" has become established, in which carefully selected and composed sentences are used to achieve a certain visual style in the synthesized image. In this note, we present an alternative approach based on retrievalaugmented diffusion models (RDMs). In RDMs, a set of nearest neighbors is retrieved from an external database during training for each training instance, and the diffusion model is conditioned on these informative samples. During inference (sampling), we replace the retrieval database with a more specialized database that contains, for example, only images of a particular visual style. This provides a novel way to "prompt" a general trained model after training and thereby specify a particular visual style. As shown by our experiments, this approach is superior to specifying the visual style within the text prompt. We open-source code and model weights at https://github.com/CompVis/latent-diffusion.

show abstract

GNN-LM: Language Modeling based on Global Contexts via GNN

Cited by 9 publications

References 53 publications

N/A Modern Approaches in Natural Language Processing

N/A Modern Approaches in Natural Language Processing

Possibilities of Using De Novo Design for Generating Diverse Functional Food Enzymes

Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Models

Contact Info

Product

Resources

About