Human Evaluation of English–Irish Transformer-Based NMT

Abstract

In this study, a human evaluation is carried out on how hyperparameter settings impact the quality of Transformer-based Neural Machine Translation (NMT) for the low-resourced English–Irish pair. SentencePiece models using both Byte Pair Encoding (BPE) and unigram approaches were appraised. Variations in model architectures included modifying the number of layers, evaluating the optimal number of heads for attention and testing various regularisation techniques. The greatest performance improvement was recorded for a Transformer-optimized model with a 16k BPE subword model. Compared with a baseline Recurrent Neural Network (RNN) model, a Transformer-optimized model demonstrated a BLEU score improvement of 7.8 points. When benchmarked against Google Translate, our translation engines demonstrated significant improvements. Furthermore, a quantitative fine-grained manual evaluation was conducted which compared the performance of machine translation systems. Using the Multidimensional Quality Metrics (MQM) error taxonomy, a human evaluation of the error types generated by an RNN-based system and a Transformer-based system was explored. Our findings show the best-performing Transformer system significantly reduces both accuracy and fluency errors when compared with an RNN-based model.

Expanding the Horizons of Machine Learning in Nanomaterials to Chiral Nanostructures

“…Vision transformers are often pre-trained on extensive datasets like ImageNet, and then subsequently fine-tuned for specialized tasks, proving efficient in areas with limited data. [ 38 ] For large language models, such as GPT and BERT, extensive pre-training on vast text corpora and low-resource languages enables deep language understanding [ 39 ] and adaptability to specialized domains in materials science. [ 40 , 41 ] Notably, large language models and multimodal vision-language models (such as GPT-4 V, LLaVA) exhibit advanced capabilities in two novel approaches, namely zero-shot and few-shot learning.…”

adaptNMT: an open-source, language-agnostic development environment for neural machine translation

“…Breakthrough performance improvements in the area of MT have been achieved through research efforts focusing on NMT (Bahdanau et al, 2014) but the advent of the Transformer architecture has greatly improved MT performance. Consequently, SOTA performance has been attained on multiple language pairs (Bojar et al, 2017(Bojar et al, , 2018Lankford et al, 2021bLankford et al, , 2022b.…”

Building Neural Machine Translation Systems for Multilingual Participatory Spaces

“…The remaining seven pairs are EN-GA, DE-GA, FR-PL, FR-GA, IT-PL, IT-GA and PL-GA; note that five out of these seven pairs involve the Irish language (GA). As of now, we prioritise corpus combinations for the other two language pairs (FR-PL and IT-PL); given our local expertise and in-house crawled datasets [35,36], all MT models involving Irish already produce high BLEU scores, and so there is less room for improvement. However, we plan to perform corpus combination for the Irish language once all the other languages are covered.…”

“…It is interesting to note that all models containing Irish as either a source or target language outperform Google Translate. While this may be unexpected in general, it was not a surprise to us; as mentioned in Section 5.3.3, we have access to many good-quality datasets for the Irish language (e.g., in the areas of health [35] and the legal domain [36]) from previous projects, and this considerably improves translation performance compared to the other language pairs, despite not being in the specific areas covered by EUComMeet. These datasets are not in the public domain, so are unavailable to Google Translate as additional training data to improve their engines.…”