UniTE: Unified Translation Evaluation

Wan, Yu; Liu, Dayiheng; Yang, Baosong; Zhang, Haibo; Chen, Boxing; Wong, Derek F.; Chao, Lidia S.

doi:10.18653/v1/2022.acl-long.558

Cited by 24 publications

(24 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Likewise, having access to only the source would correspond to sentence difficulty/complexity estimation. Similarly to Wan et al (2022); Don-Yehiya et al ( 2022), we explored both of these modes and found very high sentence-level correlations.…”

Section: Complexity and Fluency Estimationmentioning

confidence: 62%

Poor Man's Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference

Zouhar¹,

Dhuliawala²,

Zhou³

et al. 2023

Preprint

View full text Add to dashboard Cite

Machine translation quality estimation (QE) predicts human judgements of a translation hypothesis without seeing the reference. Stateof-the-art QE systems based on pretrained language models have been achieving remarkable correlations with human judgements yet they are computationally heavy and require human annotations, which are slow and expensive to create. To address these limitations, we define the problem of metric estimation (ME) where one predicts the automated metric scores also without the reference. We show that even without access to the reference, our model can estimate automated metrics (ρ=60% for BLEU, ρ=51% for other metrics) at the sentence-level. Because automated metrics correlate with human judgements, we can leverage the ME task for pre-training a QE model. For the QE task, we find that pre-training on TER is better (ρ=23%) than training for scratch (ρ=20%).

show abstract

Section: Complexity and Fluency Estimationmentioning

confidence: 62%

Poor Man's Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference

Zouhar¹,

Dhuliawala²,

Zhou³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…There are two standard methods to extract total embedding, i.e., averaging all token embeddings and using the first token embedding. (Ranasinghe et al, 2020;Wan et al, 2022) proves the superiority of using the first token embedding compared to averaging all token embeddings. Thus, we employ the final embedding of first token e f irst as the representation of unified input x.…”

Section: Unified Embeddingmentioning

confidence: 89%

“…They likewise face the same issue of poor evaluation metrics (EMs). A recent popular trend in evaluating text generation is the design of automatic EMs based on large language models (LLMs) (Zhao et al, 2019;Sellam et al, 2020;Yuan et al, 2021;Wan et al, 2022). COMET (Rei et al, 2020) and BERTScore (Zhang et al, 2020) are two typical LLM-based text EMs, where COMET provides a text EM by learning human judgments of training data and BERTScore is computed using contextualized token embeddings of BERT (Devlin et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

CodeScore: Evaluating Code Generation by Learning Code Execution

Dong¹,

Ding²,

Xue³

et al. 2023

Preprint

View full text Add to dashboard Cite

A proper code evaluation metric (CEM) profoundly impacts the evolution of code generation, which is an important research field in NLP and software engineering. Prevailing CEMs can be categorized into match-based CEMs (e.g., BLEU, Accuracy, and Code-BLEU) and execution-based CEMs (e.g., Avg-PassRatio and Pass@k), but both of them suffer from some issues. The former only measures differences in surface form regardless of the functional equivalence of codes, while the latter has huge execution overheads, including collecting expensive test cases, resolving tedious execution dependencies, and enormous execution time. To address these issues, in this paper, we propose CodeScore, an efficient and effective CEM for code generation, which estimates test case PassRatio of generated code without executing code. We also present a framework named UniCE for training unified code evaluation models by learning code execution, i.e., learning PassRatio and Executability of generated code. In order to learn code execution comprehensively, we construct more than 100 test cases for each task in several popular benchmark datasets, covering MBPP, APPS, and HumanEval. Experimental results show that CodeScore has obtained a state-of-the-art correlation with executionbased CEMs. CodeScore is strongly correlated with AvgPassPatio, and binary Code-Score is moderately correlated with Pass@1. In particular, CodeScore eliminates the need for test cases and execution dependencies in inference, and CodeScore reduces execution time by three orders of magnitude compared to AvgPassPatio and Pass@1.

show abstract

“…In order to tackle this problem, it is better to use semantic evaluation metrics such as reconstruction-BLEU , COMET (Rei et al, 2020), BLEURT (Sellam et al, 2020), UniTE (Wan et al, 2022) and so on. Here, we choose reconstruction-BLEU, which uses a reverse NMT model trained on initial single-reference corpus to translate each candidate ŷ(n) k back to a source sentence x(n) k , and then evaluates the BLEU score between…”

Section: Reward Computationmentioning

confidence: 99%

Bridging the Gap between Training and Inference: Multi-Candidate Optimization for Diverse Neural Machine Translation

Lin¹,

Yang²,

Yao³

et al. 2022

Findings of the Association for Computational Linguistics: NAACL 2022

Self Cite

View full text Add to dashboard Cite

Diverse NMT aims at generating multiple diverse yet faithful translations given a source sentence. In this paper, we investigate a common shortcoming in existing diverse NMT studies: the model is usually trained with single reference, while expected to generate multiple candidate translations in inference. The discrepancy between training and inference enlarges the confidence variance and quality gap among candidate translations and thus hinders model performance. To deal with this defect, we propose a multi-candidate optimization framework for diverse NMT. Specifically, we define assessments to score the diversity and the quality of candidate translations during training, and optimize the diverse NMT model with two strategies based on reinforcement learning, namely hard constrained training and soft constrained training. We conduct experiments on NIST Chinese-English and WMT14 English-German translation tasks. The results illustrate that our framework is transparent to basic diverse NMT models, and universally makes better trade-off between diversity and quality. Our source code is available at https://github.com/ DeepLearnXMU/MultiCanOptim.

show abstract

UniTE: Unified Translation Evaluation

Cited by 24 publications

References 4 publications

Poor Man's Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference

Poor Man's Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference

CodeScore: Evaluating Code Generation by Learning Code Execution

Bridging the Gap between Training and Inference: Multi-Candidate Optimization for Diverse Neural Machine Translation

Contact Info

Product

Resources

About