NMTScore: A Multilingual Analysis of Translation-based Text Similarity Measures

Vamvas, Jannis; Sennrich, Rico

doi:10.18653/v1/2022.findings-emnlp.15

Cited by 3 publications

(3 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They also perform extensive bitext filtering, using several different language ID tools and the filtering method proposed in which uses perplexities of a GPT-2 model (Radford et al, 2019), LAESR embeddings (Chaudhary et al, 2019, NMTScore (Vamvas and Sennrich, 2022) using Prism (Thompson and Post, 2020a,b), and WAScore (Steingrímsson et al, 2021), as well as Bicleaner AI (Zaragoza-Bernabeu et al, 2022). -Hoang et al (2023) focus on using the phrase based dictionary to distill the high-quality sentences and making a pipeline to re-ranking the top-K cosine similarity.…”

Section: Steingrímssonmentioning

confidence: 99%

Findings of the WMT 2023 Shared Task on Parallel Data Curation

Sloto,

Thompson,

Khayrallah

et al. 2023

Proceedings of the Eighth Conference on Machine Translation

View full text Add to dashboard Cite

Building upon prior WMT shared tasks in document alignment and sentence filtering, we posed the open-ended shared task of finding the best subset of possible training data from a collection of Estonian-Lithuanian web data. Participants could focus on any portion of the end-to-end data curation pipeline, including alignment and filtering. We evaluated results based on downstream machine translation quality. We release processed Common Crawl data, along with various intermediate states from a strong baseline system, which we believe will enable future research on this topic.

show abstract

Section: Steingrímssonmentioning

confidence: 99%

Findings of the WMT 2023 Shared Task on Parallel Data Curation

Sloto,

Thompson,

Khayrallah

et al. 2023

Proceedings of the Eighth Conference on Machine Translation

View full text Add to dashboard Cite

show abstract

“…For each labeled span in the source sentence, we rank all the projection candidates that share the same category as the source span using their translation probabilities (also known as translation equivalence) which have been obtained by apply-ing the pretrained M2M100 (Fan et al, 2021) or NLLB200 (Costa-jussà et al, 2022) MT models and the NMTScore library 2 (Vamvas and Sennrich, 2022). Thus, given the source span A and the candidate B the translation probability is computed as follows (Vamvas and Sennrich, 2022):…”

Section: Candidate Selectionmentioning

confidence: 99%

“…Second, we rank the candidates based on the probability of being generated as a translation of the source spans. We use the M2M100 (Fan et al, 2021) and NLLB200 (Costajussà et al, 2022) state-of-the-art MT models to compute the translation probabilities (Vamvas and Sennrich, 2022).…”

Section: Introductionmentioning

confidence: 99%

T-Projection: High Quality Annotation Projection for Sequence Labeling Tasks

García-Ferrero,

Agerri,

Rigau

2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

In the absence of readily available labeled data for a given sequence labeling task and language, annotation projection has been proposed as one of the possible strategies to automatically generate annotated data. Annotation projection has often been formulated as the task of transporting, on parallel corpora, the labels pertaining to a given span in the source language into its corresponding span in the target language. In this paper we present T-Projection, a novel approach for annotation projection that leverages large pretrained text-to-text language models and state-of-the-art machine translation technology. T-Projection decomposes the label projection task into two subtasks: (i) A candidate generation step, in which a set of projection candidates using a multilingual T5 model is generated and, (ii) a candidate selection step, in which the generated candidates are ranked based on translation probabilities. We conducted experiments on intrinsic and extrinsic tasks in 5 Indo-European and 8 lowresource African languages. We demostrate that T-projection outperforms previous annotation projection methods by a wide margin. We believe that T-Projection can help to automatically alleviate the lack of high-quality training data for sequence labeling tasks. Code and data are publicly available. 1

show abstract

A Survey on Evaluation Metrics for Machine Translation

Lee

Moon

et al. 2023

Mathematics

View full text Add to dashboard Cite

The success of Transformer architecture has seen increased interest in machine translation (MT). The translation quality of neural network-based MT transcends that of translations derived using statistical methods. This growth in MT research has entailed the development of accurate automatic evaluation metrics that allow us to track the performance of MT. However, automatically evaluating and comparing MT systems is a challenging task. Several studies have shown that traditional metrics (e.g., BLEU, TER) show poor performance in capturing semantic similarity between MT outputs and human reference translations. To date, to improve performance, various evaluation metrics have been proposed using the Transformer architecture. However, a systematic and comprehensive literature review on these metrics is still missing. Therefore, it is necessary to survey the existing automatic evaluation metrics of MT to enable both established and new researchers to quickly understand the trend of MT evaluation over the past few years. In this survey, we present the trend of automatic evaluation metrics. To better understand the developments in the field, we provide the taxonomy of the automatic evaluation metrics. Then, we explain the key contributions and shortcomings of the metrics. In addition, we select the representative metrics from the taxonomy, and conduct experiments to analyze related problems. Finally, we discuss the limitation of the current automatic metric studies through the experimentation and our suggestions for further research to improve the automatic evaluation metrics.

show abstract

NMTScore: A Multilingual Analysis of Translation-based Text Similarity Measures

Cited by 3 publications

References 15 publications

Findings of the WMT 2023 Shared Task on Parallel Data Curation

Findings of the WMT 2023 Shared Task on Parallel Data Curation

T-Projection: High Quality Annotation Projection for Sequence Labeling Tasks

A Survey on Evaluation Metrics for Machine Translation

Contact Info

Product

Resources

About