The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

Gehrmann, Sebastian; Adewumi, Tosin; Aggarwal, Karmanya; Ammanamanchi, Pawan Sasanka; Aremu, Anuoluwapo; Bosselut, Antoine; Chandu, Khyathi Raghavi; Clinciu, Miruna; Das, Dipanjan; Dhole, Kaustubh; Du, Wanyu; Durmus, Esin; Dušek, Ondřej; Emezue, Chris Chinenye; Gangal, Varun; Gârbacea, Cristina; Hashimoto, Tatsunori; Hou, Yufang; Jernite, Yacine; Jhamtani, Harsh; Ji, Yangfeng; Jolly, Shailza; Kale, Mihir; Kumar, Dharmender; Ladhak, Faisal; Madaan, Aman; Maddela, Mounica; Mahajan, Khyati; Mahamood, Saad; Majumder, Bodhisattwa Prasad; Martins, Pedro; McMillan-Major, Angelina; Mille, Simon; Miltenburg, Emiel van; Nadeem, Moin; Narayan, Shashi; Nikolaev, Vitaly; Niyongabo, Rubungo Andre; Osei, Salomey; Parikh, Ankur P.; Perez-Beltrachini, Laura; Rao, Niranjan Ramesh; Raunak, Vikas; Rodríguez, Juan Diego; Santhanam, Sashank; Sedoc, João; Sellam, Thibault; Shaikh, Samira; Shimorina, Anastasia; Cabezudo, Marco Antonio Sobrevilla; Strobelt, Hendrik; Subramani, Nishant; Xu, Wei; Yang, Diyi; Zhou, Jiawei

doi:10.48550/arxiv.2102.01672

Cited by 43 publications

(46 citation statements)

References 83 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite the advancements in deep neural networks and the demonstration of SoTA performances in NLG tasks [28,10] by a model such as DialoGPT, challenges still exist [30]. Hence, future work may involve understanding the mathematics of languages/linguistics and their relatedness.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Småprat: DialoGPT for Natural Language Generation of Swedish Dialogue by Transfer Learning

Adewumi¹,

Abid²,

Pahlavan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Building open-domain conversational systems (or chatbots) that produce convincing responses is a recognized challenge.Recent state-of-theart (SoTA) transformer-based models for the generation of natural language dialogue have demonstrated impressive performance in simulating human-like, single-turn conversations in English. This work investigates, by an empirical study, the potential for transfer learning of such models to Swedish language. DialoGPT, an English language pre-trained model, is adapted by training on three different Swedish language conversational datasets obtained from publicly available sources. Perplexity score (an automated intrinsic language model metric) and surveys by human evaluation were used to assess the performances of the fine-tuned models, with results that indicate that the capacity for transfer learning can be exploited with considerable success. Human evaluators asked to score the simulated dialogue judged over 57% of the chatbot responses to be human-like for the model trained on the largest (Swedish) dataset. We provide the demos and model checkpoints of our English and Swedish chatbots on the HuggingFace platform for public use.

show abstract

Section: Discussionmentioning

confidence: 99%

“…Advances in deep neural networks, such as the tranformer-based architectures, have brought improvements to the field [6,20,12]. These models have demonstrated SoTA performances in natural language understanding (NLU) and Natural Language Generation (NLG) tasks [28,10].…”

Section: Introductionmentioning

confidence: 99%

Småprat: DialoGPT for Natural Language Generation of Swedish Dialogue by Transfer Learning

Adewumi¹,

Abid²,

Pahlavan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Similarly, [Liu et al 2020d] proposed the General Language Generation Evaluation (GLGE), a new multi-task benchmark for natural language generation. DecaNLP [McCann et al 2018] and GEM [Gehrmann et al 2021] are also well-known benchmarks for NLG tasks.…”

Section: Informativenessmentioning

confidence: 99%

A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models

Zhang¹,

Song²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Controllable Text Generation (CTG) is emerging area in the field of natural language generation (NLG). It is regarded as crucial for the development of advanced text generation technologies that are more natural and better meet the specific constraints in practical applications. In recent years, methods using large-scale pre-trained language models (PLMs), in particular the widely used transformer-based PLMs, have become a new paradigm of NLG, allowing generation of more diverse and fluent text. However, due to the lower level of interpretability of deep neural networks, the controllability of these methods need to be guaranteed. To this end, controllable text generation using transformer-based PLMs has become a rapidly growing yet challenging new research hotspot. A diverse range of approaches have emerged in the recent 3-4 years, targeting different CTG tasks which may require different types of controlled constraints. In this paper, we present a systematic critical review on the common tasks, main approaches and evaluation methods in this area. Finally, we discuss the challenges that the field is facing, and put forward various promising future directions. To the best of our knowledge, this is the first survey paper to summarize CTG techniques from the perspective of PLMs. We hope it can help researchers in related fields to quickly track the academic frontier, providing them with a landscape of the area and a roadmap for future research.

show abstract

“…Large, sufficiently diverse/representative, public benchmarks have spurred significant progress in tabular AutoML [14,15,20,59] and NLP [18,34,39,54]. However we are not aware of any analogous benchmarks for evaluating multimodal text/tabular ML.…”

Section: Related Workmentioning

confidence: 99%

Benchmarking Multimodal AutoML for Tabular Data with Text Fields

Shi¹,

Mueller²,

Erickson³

et al. 2021

Preprint

View full text Add to dashboard Cite

We consider the use of automated supervised learning systems for data tables that not only contain numeric/categorical columns, but one or more text fields as well.Here we assemble 18 multimodal data tables that each contain some text fields and stem from a real business application. Our publicly-available benchmark 2 enables researchers to comprehensively evaluate their own methods for supervised learning with numeric, categorical, and text features. To ensure that any single modeling strategy which performs well over all 18 datasets will serve as a practical foundation for multimodal text/tabular AutoML, the diverse datasets in our benchmark vary greatly in: sample size, problem types (a mix of classification and regression tasks), number of features (with the number of text columns ranging from 1 to 28 between datasets), as well as how the predictive signal is decomposed between text vs. numeric/categorical features (and predictive interactions thereof). Over this benchmark, we evaluate various straightforward pipelines to model such data, including standard two-stage approaches where NLP is used to featurize the text such that AutoML for tabular data can then be applied. Compared with human data science teams, the fully automated methodology 3 that performed best on our benchmark (stack ensembling a multimodal Transformer with various tree models) also manages to rank 1st place when fit to the raw text/tabular data in two MachineHack prediction competitions and 2nd place (out of 2380 teams) in Kaggle's Mercari Price Suggestion Challenge.

show abstract

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

Cited by 43 publications

References 83 publications

Småprat: DialoGPT for Natural Language Generation of Swedish Dialogue by Transfer Learning

Småprat: DialoGPT for Natural Language Generation of Swedish Dialogue by Transfer Learning

A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models

Benchmarking Multimodal AutoML for Tabular Data with Text Fields

Contact Info

Product

Resources

About