Towards the use of entropy as a measure for the reliability of automatic MT evaluation metrics

Munk, Michal; Munková, Daša; Benko, Ľubomír

doi:10.3233/jifs-169505

Cited by 13 publications

(12 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The neural machine translation achieved a better translation quality due to the reference. Its translation was more accurate in both meaning and fluency; for example: (7) Source: Michel called on residents to "stay calm and cool-headed" as the investigation continued into Tuesday's police raid. mt@ec_SMT: Michel vyzval, aby zachovali pokoj a "cool-headed"ako prešetrovanie pokračovalo utorkového policajným zásahom.…”

Section: Discussionmentioning

confidence: 99%

“…They measure the closeness and/or compute the score of their lexical concordance. According to the criterion of lexical concordance, automatic metrics of MT evaluation can be divided into metrics of accuracy (matched n-grams) and metrics of error rate (edit distance) [7].…”

Section: Introductionmentioning

confidence: 99%

“…BLEU (Bilingual Evaluation Understudy) [14], which is used in our research, is a geometric mean of n-gram precisions (p = A/B), and the second part is a brevity penalty (BP), i.e., a length-based penalty to prevent very short sentences as compensation for inappropriate translation. BLEU represents two features of MT quality-adequacy and fluency [7].…”

mentioning

confidence: 99%

See 2 more Smart Citations

Evaluation of English–Slovak Neural and Statistical Machine Translation

et al. 2021

Self Cite

View full text Add to dashboard Cite

This study is focused on the comparison of phrase-based statistical machine translation (SMT) systems and neural machine translation (NMT) systems using automatic metrics for translation quality evaluation for the language pair of English and Slovak. As the statistical approach is the predecessor of neural machine translation, it was assumed that the neural network approach would generate results with a better quality. An experiment was performed using residuals to compare the scores of automatic metrics of the accuracy (BLEU_n) of the statistical machine translation with those of the neural machine translation. The results showed that the assumption of better neural machine translation quality regardless of the system used was confirmed. There were statistically significant differences between the SMT and NMT in favor of the NMT based on all BLEU_n scores. The neural machine translation achieved a better quality of translation of journalistic texts from English into Slovak, regardless of if it was a system trained on general texts, such as Google Translate, or specific ones, such as the European Commission’s (EC’s) tool, which was trained on a specific-domain.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Evaluation of English–Slovak Neural and Statistical Machine Translation

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…The tags are of unequal length, but most tags follow the same structure for the same inflectional paradigm. The information of the part of speech is often more important than the structure or the length of the tags [26,27] and it is encoded in the first position of every tag. The second position usually marks an inflectional paradigm for words that have this category.…”

Section: Methodsmentioning

confidence: 99%

Improvement of Misleading and Fake News Classification for Flective Languages by Morphological Group Analysis

Kapusta

Obonya

2020

Informatics

View full text Add to dashboard Cite

Due to the constantly evolving social media and different types of sources of information, we are facing different fake news and different types of misinformation. Currently, we are working on a project to identify applicable methods for identifying fake news for floating language types. We explored different approaches to detect fake news in the presented research, which are based on morphological analysis. This is one of the basic components of natural language processing. The aim of the article is to find out whether it is possible to improve the methods of dataset preparation based on morphological analysis. We collected our own and unique dataset, which consisted of articles from verified publishers and articles from news portals that are known as the publishers of fake and misleading news. Articles were in the Slovak language, which belongs to the floating types of languages. We explored different approaches in this article to the dataset preparation based on morphological analysis. The prepared datasets were the input data for creating the classifier of fake and real news. We selected decision trees for classification. The evaluation of the success of two different methods of preparation was carried out because of the success of the created classifier. We found a suitable dataset pre-processing technique by morphological group analysis. This technique could be used for improving fake news classification.

show abstract

“…We focused on automatic metrics, but we also used the post-editing method, which is a manual evaluation of MT quality. According to the criterion of concordance, we selected automatic metrics based on lexical concordance and divided them into metrics of accuracy and metrics of error rate (Munk, Munkova & Benko, 2018).…”

Section: Automatic Evaluation Metricsmentioning

confidence: 99%

The role of automated evaluation techniques in online professional translator training

Munková

Munk

Benko

et al. 2021

PeerJ Computer Science

Self Cite

View full text Add to dashboard Cite

The rapid technologisation of translation has influenced the translation industry’s direction towards machine translation, post-editing, subtitling services and video content translation. Besides, the pandemic situation associated with COVID-19 has rapidly increased the transfer of business and education to the virtual world. This situation has motivated us not only to look for new approaches to online translator training, which requires a different method than learning foreign languages but in particular to look for new approaches to assess translator performance within online educational environments. Translation quality assessment is a key task, as the concept of quality is closely linked to the concept of optimization. Automatic metrics are very good indicators of quality, but they do not provide sufficient and detailed linguistic information about translations or post-edited machine translations. However, using their residuals, we can identify the segments with the largest distances between the post-edited machine translations and machine translations, which allow us to focus on a more detailed textual analysis of suspicious segments. We introduce a unique online teaching and learning system, which is specifically “tailored” for online translators’ training and subsequently we focus on a new approach to assess translators’ competences using evaluation techniques—the metrics of automatic evaluation and their residuals. We show that the residuals of the metrics of accuracy (BLEU_n) and error rate (PER, WER, TER, CDER, and HTER) for machine translation post-editing are valid for translator assessment. Using the residuals of the metrics of accuracy and error rate, we can identify errors in post-editing (critical, major, and minor) and subsequently utilize them in more detailed linguistic analysis.

show abstract

Towards the use of entropy as a measure for the reliability of automatic MT evaluation metrics

Cited by 13 publications

References 16 publications

Evaluation of English–Slovak Neural and Statistical Machine Translation

Evaluation of English–Slovak Neural and Statistical Machine Translation

Improvement of Misleading and Fake News Classification for Flective Languages by Morphological Group Analysis

The role of automated evaluation techniques in online professional translator training

Contact Info

Product

Resources

About