Neural Generation for Czech: Data and Baselines

Dušek, Ondřej; Jurčíček, Filip

doi:10.18653/v1/w19-8670

Cited by 13 publications

(8 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Communicative Goal Language(s) Size Input Type CommonGEN Produce a likely sentence which mentions all of the source concepts. en 67k Concept Set Czech Restaurant (Dušek and Jurčíček, 2019) Produce a text expressing the given intent and covering the specified attributes. (NLU) tasks.…”

Section: Datasetmentioning

confidence: 99%

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

Gehrmann¹,

Adewumi²,

Aggarwal³

et al. 2021

Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)

Self Cite

View full text Add to dashboard Cite

We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with wellestablished, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for which we are organizing a shared task at our ACL 2021 Workshop and to which we invite the entire NLG community to participate.

show abstract

Section: Datasetmentioning

confidence: 99%

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

Gehrmann¹,

Adewumi²,

Aggarwal³

et al. 2021

Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Training data for NLG in languages other than English is still very limited: there are small datasets in Korean (Chen et al, 2010), Spanish (García-Méndez et al, 2019), and Czech (Dušek and Jurčíček, 2019). There exist also structured data-to-text datasets for German and French (Nema et al, 2018) and image-todescription datasets in Chinese (Li et al, 2016c) and Dutch (van Miltenburg et al, 2017, as well as cross-lingual English-German data (Elliott et al, 2016).…”

Section: Available Datasetsmentioning

confidence: 99%

Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems

Razumovskaia¹,

Glavaš²,

Majewska³

et al. 2021

Preprint

View full text Add to dashboard Cite

Despite the fact that natural language conversations with machines represent one of the central objectives of AI, and despite the massive increase of research and development efforts in conversational AI, task-oriented dialogue (TOD) -i.e., conversations with an artificial agent with the aim of completing a concrete task -is currently limited to a few narrow domains (e.g., food ordering, ticket booking) and a handful of major languages (e.g., English, Chinese). In this work, we provide an extensive overview of existing efforts in multilingual TOD and analyse the factors preventing the development of truly multilingual TOD systems. We identify two main challenges that combined hinder the faster progress in multilingual TOD: (1) current state-of-the-art TOD models based on large pretrained neural language models are data hungry; at the same time (2) data acquisition for TOD use cases is expensive and tedious. Most existing approaches to multilingual TOD thus rely on (zero-or few-shot) cross-lingual transfer from resource-rich languages (in TOD, this is basically only English), either by means of (i) machine translation or (ii) multilingual representation spaces. However, such approaches are currently not a viable solution for a large number of low-resource languages without parallel data and/or limited monolingual corpora. Finally, we discuss critical challenges and potential solutions by drawing parallels between TOD and other cross-lingual and multilingual NLP research.

show abstract

“…Most recently, Gehrmann et al [22] proposed GEM, a benchmark specifically for tasks requiring Natural Language Generation. GEM encompasses 4 tasks through 11 datasets: Summarization [72,49,19], Structure To Text [20,40,17,57,16,54], Dialogue [63], and Simplification [25,85,1]. As noted in the introduction, evaluation of generated language relies not only on tasks, but also on automatic metrics (or humans).…”

Section: Benchmarksmentioning

confidence: 99%

BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation

Scialom,

Hill

2021

Preprint

View full text Add to dashboard Cite

Natural language processing (NLP) systems are increasingly trained to generate open-ended text rather than classifying between responses. This makes research on evaluation metrics for generated language -functions that score system output given the context and/or human reference responses -of critical importance. However, different metrics have different strengths and biases, and reflect human intuitions better on some tasks than others. There is currently no simple, unified way to compare, analyse or evaluate metrics across a representative set of tasks. Here, we describe the Benchmark to Evaluate Automatic Metrics (BEAMetrics), a resource to make research into new metrics itself easier to evaluate. BEAMetrics users can quickly compare existing and new metrics with human judgements across a diverse set of tasks, quality dimensions (fluency vs. coherence vs. informativeness etc), and languages. As generation experts might predict, BEAMetrics reveals stark taskdependent differences between existing metrics, and consistently poor performance on tasks with complex answer spaces or high reliance on general knowledge. While this analysis highlights a critical issue facing current research practice, BEAMetrics also contribute to its resolution by facilitating research into better metrics -particularly those that can account for the complex interaction between context and general knowledge inherent to many modern NLP applications. 1

show abstract

Neural Generation for Czech: Data and Baselines

Cited by 13 publications

References 65 publications

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems

BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation

Contact Info

Product

Resources

About