Offline RL for Natural Language Generation with Implicit Language Q Learning

Snell, Charlie; Kostrikov, Ilya; Su, Yi; Yang, Ming‐Bo; Levine, Sergey

doi:10.48550/arxiv.2206.11871

Cited by 7 publications

(15 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Reinforcement Learning from Feedback RL has been applied to enhance various models in NLP tasks such as machine translation [117], summarization [18], dialogue generation [118], image captioning [119], question generation [120], text-games [121], and more [122,123,124]. RL is a helpful method for optimizing non-differentiable objectives in language generation tasks by treating them as sequential decisionmaking problems.…”

Section: Instruction-aligning Methodsmentioning

confidence: 99%

A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT

Zhou¹,

Li²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

The Pretrained Foundation Models (PFMs) are regarded as the foundation for various downstream tasks with different data modalities. A pretrained foundation model, such as BERT, GPT-3, MAE, DALLE-E, and ChatGPT, is trained on large-scale data which provides a reasonable parameter initialization for a wide range of downstream applications. The idea of pretraining behind PFMs plays an important role in the application of large models. As a type of transfer learning paradigm, pretraining is applied in computer vision with frozen and fine-tuning techniques, showing promising performance. Word embedding in the natural language process can be also regarded as a type of pertaining, but it suffers many problems such as polysemy. Different from previous methods that apply convolution and recurrent modules for feature extractions, the generative pre-training (GPT) method applies Transformer as the feature extractor and is trained on large datasets with an autoregressive paradigm. Similarly, the BERT apples transformers to train on large datasets as a contextual language model. Recently, the ChatGPT shows promising success on large language models, which applies an autoregressive language model with zero shot or few show prompting. With the extraordinary success of PFMs, AI has made waves in a variety of fields over the past few years. Considerable methods, datasets, and evaluation metrics have been proposed in the literature, the need is raising for an updated survey.This study provides a comprehensive review of recent research advancements, current and future challenges, and opportunities for PFMs in text, image, graph, as well as other data modalities. We first review the basic components and existing pretraining in natural language processing, computer vision, and graph learning. We then discuss other advanced PFMs for other data modalities and unified PFMs considering the data quality and quantity. Besides, we discuss relevant research about the fundamentals of the PFM, including model efficiency and compression, security, and privacy. Finally, we lay out key implications, future research directions, challenges, and open problems. We hope this survey can shed light on the research of PFMs on the scalability, reasoning ability, cross-domain ability, user-friendly interactive ability, security and privacy-preserving ability for artificial general intelligence.

show abstract

Section: Instruction-aligning Methodsmentioning

confidence: 99%

A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT

Zhou¹,

Li²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…On the other hand, Offline RL (Fujimoto et al, 2019;Kumar et al, 2020;Brandfonbrener et al, 2021;Kostrikov et al, 2021) removes all need for environment interaction or user simulators, instead operating purely on static datasets of prior human interaction. There are many closely related works (Jaques et al, 2019(Jaques et al, , 2020Snell et al, 2022;Cohen et al, 2022;Verma et al, 2022;Jang et al, 2022) based on offline RL that policy improvement via behavior cloning of self-generated utterances, which inherits the ability of pre-trained language models to generate human-like responses. In RL parlance, such methods could be considered policy extraction with approximate dynamic programming.…”

Section: Related Workmentioning

confidence: 99%

Deep RL with Hierarchical Action Exploration for Dialogue Generation

Cho¹,

Takahashi²,

Yanase³

et al. 2023

Preprint

View full text Add to dashboard Cite

Conventionally, since the natural language action space is astronomical, approximate dynamic programming applied to dialogue generation involves policy improvement with action sampling. However, such a practice is inefficient for reinforcement learning (RL) because the eligible (high action value) responses are very sparse, and the greedy policy sustained by the random sampling is flabby. This paper shows that the performance of dialogue policy positively correlated with sampling size by theoretical and experimental. We introduce a novel dual-granularity Q-function to alleviate this limitation by exploring the most promising response category to intervene in the sampling. It extracts the actions following the grained hierarchy, which can achieve the optimum with fewer policy iterations. Our approach learns in the way of offline RL from multiple reward functions designed to recognize human emotional details. Empirical studies demonstrate that our algorithm outperforms the baseline methods. Further verification presents that ours can generate responses with higher expected rewards and controllability.

show abstract

“…Drawing inspiration from the work of [32], we formalize our problem as a Partially Observable Markov Decision Process [33]. At each timestep t, the agent (i.e.…”

Section: Rl Task Formulationmentioning

confidence: 99%

Offline RL for generative design of protein binders

Tarasov,

Mbou Sob,

Arbesú

et al. 2023

Preprint

View full text Add to dashboard Cite

Offline Reinforcement Learning (RL) offers a compelling avenue for solving RL problems without the need for interactions with an environment, which may be expensive or unsafe. While online RL methods have found success in various domains, such asde novoStructure-Based Drug Discovery (SBDD), they struggle when it comes to optimizing essential properties derived from protein-ligand docking. The high computational cost associated with the docking process makes it impractical for online RL, which typically requires hundreds of thousands of interactions during learning. In this study, we propose the application of offline RL to address the bottleneck posed by the docking process, leveraging RL’s capability to optimize non-differentiable properties. Our preliminary investigation focuses on using offline RL to conditionally generate drugs with improved docking and chemical properties.

show abstract

Offline RL for Natural Language Generation with Implicit Language Q Learning

Cited by 7 publications

References 0 publications

A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT

A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT

Deep RL with Hierarchical Action Exploration for Dialogue Generation

Offline RL for generative design of protein binders

Contact Info

Product

Resources

About