No abstract

We present LaMDA: Language Models for Dialog Applications. LaMDA is a family of Transformerbased neural language models specialized for dialog, which have up to 137B parameters and are pre-trained on 1.56T words of public dialog data and web text. While model scaling alone can improve quality, it shows less improvements on safety and factual grounding. We demonstrate that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding. The first challenge, safety, involves ensuring that the model's responses are consistent with a set of human values, such as preventing harmful suggestions and unfair bias. We quantify safety using a metric based on an illustrative set of human values, and we find that filtering candidate responses using a LaMDA classifier fine-tuned with a small amount of crowdworker-annotated data offers a promising approach to improving model safety. The second challenge, factual grounding, involves enabling the model to consult external knowledge sources, such as an information retrieval system, a language translator, and a calculator. We quantify factuality using a groundedness metric, and we find that our approach enables the model to generate responses grounded in known sources, rather than responses that merely sound plausible. Finally, we explore the use of LaMDA in the domains of education and content recommendations, and analyze their helpfulness and role consistency. * Work done while at Google.

In previous work, researchers have

No abstract

Learning e ective feature crosses is the key behind building recommender systems. However, the sparse and large feature space requires exhaustive search to identify e ective crosses. Deep & Cross Network (DCN) was proposed to automatically and e ciently learn bounded-degree predictive feature interactions. Unfortunately, in models that serve web-scale tra c with billions of training examples, DCN showed limited expressiveness in its cross network at learning more predictive feature interactions. Despite signi cant research progress made, many deep learning models in production still rely on traditional feed-forward neural networks to learn feature crosses ine ciently.In light of the pros/cons of DCN and existing feature interaction learning approaches, we propose an improved framework DCN-M to make DCN more practical in large-scale industrial se ings. In a comprehensive experimental study with extensive hyper-parameter search and model tuning, we observed that DCN-M approaches outperform all the state-of-the-art algorithms on popular benchmark datasets. e improved DCN-M is more expressive yet remains cost e cient at feature interaction learning, especially when coupled with a mixture of low-rank architecture. DCN-M is simple, can be easily adopted as building blocks, and has delivered signi cant o ine accuracy and online business metrics gains across many web-scale learning to rank systems.