An Evaluation of the Accuracy of the Machine Translation Systems of Social Media Language

Abstract

In this age of information technology, it has become possible for people all over the world to communicate in different languages through social media platforms with the help of machine translation (MT) systems. As far as the Arabic-English language pair is concerned, most studies have been conducted on evaluating the MT output for the standard varieties of Arabic, with fewer studies focusing on the vernacular or colloquial varieties. This study attempts to address this gap through presenting an evaluation of the performance of MT output for vernacular or colloquial Arabic in the social media domain. As it is currently the most widely used MT system, Google Translate (GT) has been chosen for evaluating the reliability of its output in the context of translating the Arabic colloquial language (i.e., Egyptian/Cairene Arabic variety) used in social media into English. With this goal in mind, a corpus consisting of Egyptian dialectal Arabic sentences were collected from social media networks, i.e., Facebook and Twitter, and then fed into GT system. The GT output was then evaluated by three human translators to assess their accuracy of translation in terms of adequacy and fluency. The results of the study show that several translation problems have been spotted for GT output. These problems are mainly concerned with wrong equivalents, inappropriate additions and deletions, and transliteration for out-of-vocabulary (OOV) words, which are mostly due to the literal translation of the Arabic vernacular sentences into English. This can be due to the fact that Arabic vernacular varieties are different from the standard language for which MT systems have been basically developed. This, consequently, necessitates the need to upgrade such MT systems to deal with the vernacular varieties.

The Interaction Between Morphosyntactic Features and the Performance of Machine Translation Tools: The Case of Google Translate, Systran, and Microsoft Bing in English-Arabic Translation

“…Aljerf (2021) evaluates the accuracy of GT in the English-Arabic translation of technical terms, and concludes that GT faces problems with words with varying prefixes, roots combined with the same suffix, compounds and blends. Sabtan et al (2021) use comments made on social media to create a corpus of Egyptian Arabic. Using a number of sentences from that corpus, the authors feed a number of sentences into GT and evaluate the output of GT English-Arabic translation.…”

Systematic Review of English/Arabic Machine Translation Postediting: Implications for AI Application in Translation Research and Pedagogy

“…Evaluative studies stressed the significance of translation evaluation to improve the performance of Arabic MT engines and suggest strategies to overcome relevant shortcomings [55,61,65]. Considering the limited research on Arabic MT effectiveness in dealing with specialized content [100], some studies addressed MT shortcomings in translating specialized texts or textual components such as the legal discourse [100], literary features [66], proverbs [67], sentiment words [101], relative clauses [57], as well as social media vernacular [102]. Evaluative studies continued to dominate the literature on MT systems to highlight their affordances and limitations in terms of adequacy, accuracy, fluency, context sensitivity, terminology and other criteria and called for complementarity between MT and HT via pre-editing and postediting [50,53,58,63,70,103].…”

Exploring the Role of Machine Translation in Translating English Collocations into Arabic: Insights from Student Translators

“…GT, as pointed out by Alqudsi et al (2014), supports over 55 different languages. GT is currently the most widely used MT system for the English-Arabic language pair (Sabtan et al 2021).…”