Deep AM-FM: Toolkit for Automatic Dialogue Evaluation

Zhang, Chen; D’Haro, Luis Fernando; Banchs, Rafael E.; Friedrichs, Thomas; Li, Haizhou

doi:10.1007/978-981-15-8395-7_5

Cited by 27 publications

(32 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the distribution of the number of constraint tokens in the experiments of these papers was much smaller than that of this task, and we found these methods did not perform well on this task. and Chen et al (2021) proposed lexically constrained decoding given explicit alignment guidance between the constraints and the source text. Alignments were induced from an additional alignment head or attention weights (Garg et al, 2019), but these methods assumed that gold alignments are given as constraints.…”

Section: Discussionmentioning

confidence: 99%

Proceedings of the 8th Workshop on Asian Translation (WAT2021)

Nakazawa¹,

Goto²

2021

View full text Add to dashboard Cite

Many Asian countries are rapidly growing these days and the importance of communicating and exchanging the information with these countries has intensified. To satisfy the demand for communication among these countries, machine translation technology is essential.Machine translation technology has rapidly evolved recently and it is seeing practical use especially between European languages. However, the translation quality of Asian languages is not that high compared to that of European languages, and machine translation technology for these languages has not reached a stage of proliferation yet. This is not only due to the lack of the language resources for Asian languages but also due to the lack of techniques to correctly transfer the meaning of sentences from/to Asian languages. Consequently, a place for gathering and sharing the resources and knowledge about Asian language translation is necessary to enhance machine translation research for Asian languages.

show abstract

Section: Discussionmentioning

confidence: 99%

Proceedings of the 8th Workshop on Asian Translation (WAT2021)

Nakazawa¹,

Goto²

2021

View full text Add to dashboard Cite

show abstract

“…Semantic appropriateness seeks to determine whether the response topically fit into its corresponding dialogue context. Many existing evaluation frameworks have focused on the two above-mentioned aspects [18]- [20]. Context coherence is concerned with the structure of the dialogue flow, such that the passage is expressed fluidly, clearly, and with sensible turn-taking [21].…”

Section: A Aspects Of Evaluationmentioning

confidence: 99%

“…For evaluation datasets from DSTC shared tasks, we compare D-score against referencebased baselines adopted by the task organizers. On the DSTC6 evaluation dataset, we also compare D-score against two more recently proposed reference-based metrics, AM-FM [19] and Deep AM-FM [18]. In addition, four state-of-the-art referencefree baselines, BERT NLI (BNLI) [11], contextualized RU-BER (Ctr-R) [7], GPT-2 [20] and USR [8], are included.…”

Section: B Baselinesmentioning

confidence: 99%

“…We consider four high-level linguistic properties, language fluency, context coherence, logical consistency and semantic appropriateness. They are identified as the properties that highly correlate with human's perception of what constitutes a good conversation [15]- [18]. We hypothesize several relevant tasks pertaining to these linguistic properties that are naturally complementary to the task of dialogue evaluation with each account for a single evaluation aspect.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

D-Score: Holistic Dialogue Evaluation Without Reference

Zhang

Lee

D’Haro

et al. 2021

IEEE/ACM Trans. Audio Speech Lang. Process.

Self Cite

View full text Add to dashboard Cite

In artistic gymnastics, difficulty score or D-score is used for judging performance. Starting from zero, an athlete earns points from different aspects such as composition requirement, difficulty, and connection between moves. The final score is a composition of the quality of various performance indicators. Similarly, when evaluating dialogue responses, human judges generally follow a number of criteria, among which language fluency, context coherence, logical consistency, and semantic appropriateness are on top of the agenda. In this paper, we propose an automatic dialogue evaluation framework called D-score that resembles the way gymnastics is evaluated. Following the four human judging criteria above, we devise a range of evaluation tasks and model them under a multi-task learning framework. The proposed framework, without relying on any human-written reference, learns to appreciate the overall quality of human-human conversations through a representation that is shared by all tasks without over-fitting to individual task domain. We evaluate D-score by performing comprehensive correlation analyses with human judgement on three dialogue evaluation datasets, among which two are from past DSTC series, and benchmark against state-of-the-art baselines. D-score not only outperforms the best baseline by a large margin in terms of system-level Spearman correlation but also represents an important step towards explainable dialogue scoring.

show abstract

“…For evaluation, as we have mentioned before, we use BLEU (Papineni et al, 2002) as the primary evaluation metric. WAT also uses metrics such as RIBES (Isozaki et al, 2010), AM-FM (Zhang et al, 2021) and human evaluation (Nakazawa et al, 2019(Nakazawa et al, , 2020(Nakazawa et al, , 2021. All these metrics focus on different aspects of translations and may lead to different rankings for submissions, however this multi-metric evaluation helps us understand that there may not be one perfect model.…”

Section: Training and Evaluationmentioning

confidence: 99%

NICT-5’s Submission To WAT 2021: MBART Pre-training And In-Domain Fine Tuning For Indic Languages

Dabre¹,

Chakrabarty²

2021

Proceedings of the 8th Workshop on Asian Translation (WAT2021)

View full text Add to dashboard Cite

In this paper we describe our submission to the multilingual Indic language translation task "MultiIndicMT" under the team name "NICT-5". This task involves translation from 10 Indic languages into English and vice-versa. The objective of the task was to explore the utility of multilingual approaches using a variety of in-domain and out-of-domain parallel and monolingual corpora. Given the recent success of multilingual NMT pre-training we decided to explore pre-training an MBART model on a large monolingual corpus collection covering all languages in this task followed by multilingual fine-tuning on small in-domain corpora. Firstly, we observed that a small amount of pretraining followed by fine-tuning on small bilingual corpora can yield large gains over when pre-training is not used. Furthermore, multilingual fine-tuning leads to further gains in translation quality which significantly outperforms a very strong multilingual baseline that does not rely on any pre-training.

show abstract

Deep AM-FM: Toolkit for Automatic Dialogue Evaluation

Cited by 27 publications

References 14 publications

Proceedings of the 8th Workshop on Asian Translation (WAT2021)

Proceedings of the 8th Workshop on Asian Translation (WAT2021)

D-Score: Holistic Dialogue Evaluation Without Reference

NICT-5’s Submission To WAT 2021: MBART Pre-training And In-Domain Fine Tuning For Indic Languages

Contact Info

Product

Resources

About