Recipes for building an open-domain chatbot

Roller, Stephen; Dinan, Emily; Goyal, Naman; Ju, Da Young; Liu, Yinhan; Xu, Jing; Ott, Myle; Shuster, Kurt; Smith, Eric M.; Boureau, Y-Lan; Weston, Jason

doi:10.48550/arxiv.2004.13637

Cited by 128 publications

(259 citation statements)

References 0 publications

Supporting

Mentioning

216

Contrasting

Order By: Relevance

“…Our work is also closely related to recent successes in applying language models to dialog modeling (e.g., [25,26,17,18]), which built on earlier research in neural dialog modeling (e.g., [14,15,16,27,28]). One of our fine-tuning stages requires training on dialog-only data, which is related to Wolf et al [29], Dinan et al [25] and Zhang et al [30].…”

Section: Related Workmentioning

confidence: 82%

“…One of our fine-tuning stages requires training on dialog-only data, which is related to Wolf et al [29], Dinan et al [25] and Zhang et al [30]. Our use of fine-tuning on crowdworker-annotated data to improve interestingness is comparable to Roller et al [18]. However, we aim to maximize the interestingness of the model's output distinctly from its ability to engage the user in further interaction.…”

Section: Related Workmentioning

confidence: 99%

“…Dialog models [14,15,16], one of the most interesting applications of large language models, successfully take advantage of Transformers' ability to represent long-term dependencies in text [17,18]. Similar to general language models [13], Adiwardana et al [17] show that dialog models are also well suited to model scaling.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

LaMDA: Language Models for Dialog Applications

Thoppilan¹,

Freitas²,

Hall³

et al. 2022

Preprint

288

270

View full text Add to dashboard Cite

We present LaMDA: Language Models for Dialog Applications. LaMDA is a family of Transformerbased neural language models specialized for dialog, which have up to 137B parameters and are pre-trained on 1.56T words of public dialog data and web text. While model scaling alone can improve quality, it shows less improvements on safety and factual grounding. We demonstrate that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding. The first challenge, safety, involves ensuring that the model's responses are consistent with a set of human values, such as preventing harmful suggestions and unfair bias. We quantify safety using a metric based on an illustrative set of human values, and we find that filtering candidate responses using a LaMDA classifier fine-tuned with a small amount of crowdworker-annotated data offers a promising approach to improving model safety. The second challenge, factual grounding, involves enabling the model to consult external knowledge sources, such as an information retrieval system, a language translator, and a calculator. We quantify factuality using a groundedness metric, and we find that our approach enables the model to generate responses grounded in known sources, rather than responses that merely sound plausible. Finally, we explore the use of LaMDA in the domains of education and content recommendations, and analyze their helpfulness and role consistency. * Work done while at Google.

show abstract

Section: Related Workmentioning

confidence: 82%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

LaMDA: Language Models for Dialog Applications

Thoppilan¹,

Freitas²,

Hall³

et al. 2022

Preprint

288

270

View full text Add to dashboard Cite

show abstract

“…Adiwardana et al (2020) proposed Meena, a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations. Blender (Roller et al, 2020;Xu et al, 2021) learn to provide engaging talking points and listen to their partners, as well as displaying knowledge, empathy and personality appropriately, while maintaining a consistent persona. Adapter-bot (Madotto et al, 2021) explored prompt-based few-shot learning in dialogue tasks.…”

Section: Related Workmentioning

confidence: 99%

“…Open-domain dialogue is a long-standing problem in natural language processing and has aroused the widespread interest of researchers. Many approaches have been studied, and recently, generation models trained on large-scale data have gained more attention (Adiwardana et al, 2020;Roller et al, 2020;Xu et al, 2021;Madotto et al, 2021;Bao et al, 2019Bao et al, , 2020Zhang et al, 2019;. Open-domain dialogue systems are born to deal with many diverse domains, and naturally its training data, usually crawled from online * Corresponding Author resources such as Reddit and Twitter, are heterogeneous and contain utterances with many various topics, more freedom of topic shifting, and vague responses (Kummerfeld et al, 2018).…”

Section: Introductionmentioning

confidence: 99%

Pan More Gold from the Sand: Refining Open-domain Dialogue Training with Noisy Self-Retrieval Generation

Wang¹,

Li²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Real human conversation data are complicated, heterogeneous, and noisy, from whom building open-domain dialogue systems remains a challenging task. In fact, such dialogue data can still contain a wealth of information and knowledge, however, they are not fully explored. In this paper, we show existing open-domain dialogue generation methods by memorizing context-response paired data with causal or encode-decode language models underutilize the training data. Different from current approaches, using external knowledge, we explore a retrieval-generation training framework that can increase the usage of training data by directly considering the heterogeneous and noisy training data as the "evidence". Experiments over publicly available datasets demonstrate that our method can help models generate better responses, even such training data are usually impressed as low-quality data. Such performance gain is comparable with those improved by enlarging the training set, even better. We also found that the model performance has a positive correlation with the relevance of the retrieved evidence. Moreover, our method performed well on zero-shot experiments, which indicates that our method can be more robust to real-world data.

show abstract

Si‐Based Dual‐Gate Field‐Effect Transistor Array for Low‐Power On‐Chip Trainable Hardware Neural Networks

Lee,

Kwon,

Lee

et al. 2023

Advanced Intelligent Systems

View full text Add to dashboard Cite

Herein, dual‐gate field‐effect transistors (DG FETs) fabricated on Si substrate and a corresponding NOR‐type array designed for low‐power on‐chip trainable hardware neural networks (HNNs) are presented. The fabricated DG FET exhibits notable endurance characteristics, with the subthreshold swing remaining consistently within a 2.45% range of change and ΔV th per cycle maintaining stability at 4.5% over repetitive program and erase operations, up to 104 cycles. Furthermore, a multilevel characteristic is achieved through low‐power program/erase operations based on Fowler–Nordheim (FN) tunneling, which exhibit 0.09 and 0.99 fJ per spike, respectively. These characteristics provide the HNN stability, along with high performance and power efficiency. The NOR‐type array in this work demonstrates selective update and bidirectional vector‐by‐matrix multiplication capabilities. This enables on‐chip training based on a gradient descent algorithm, without the need for an additional array for backpropagation. Subsequently, a simulation of the Modified National Institute of Standards and Technology classification is conducted to evaluate the accuracy and training power consumption of the proposed device in comparison to other two‐terminal memristor devices. The results show that the DG FET array achieves superior accuracy while maintaining over 180.4 times higher energy efficiency, demonstrating the potential of the DG FET as a promising candidate for low‐power HNN applications.

show abstract

Recipes for building an open-domain chatbot

Cited by 128 publications

References 0 publications

LaMDA: Language Models for Dialog Applications

LaMDA: Language Models for Dialog Applications

Pan More Gold from the Sand: Refining Open-domain Dialogue Training with Noisy Self-Retrieval Generation

Si‐Based Dual‐Gate Field‐Effect Transistor Array for Low‐Power On‐Chip Trainable Hardware Neural Networks

Contact Info

Product

Resources

About