Sentence Level Dialect Identification for Machine Translation System Selection

Salloum, Wael; Elfardy, Heba; Alamir-Salloum, Linda; Habash, Nizar; Diab, Mona

doi:10.3115/v1/p14-2125

Cited by 26 publications

(24 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(1) MSA and dialectal Arabic; (2) four-way classification -MSA, Egyptian Arabic, Gulf Arabic, and Levantine Arabic; and (3) three-way classification -Egyptian Arabic, Gulf Arabic, and Levantine Arabic. Salloum et al (2014) explores the use of sentence-level Arabic dialect identification as a pre-processor for MT, in customizing the selection of the MT model used to translate a given sentence to the dialect it uses. In performing dialect-specific MT, the authors achieve an improvement of 1.0% BLEU score compared with a baseline system which does not differentiate between Arabic dialects.…”

Section: Similar Languages Language Varieties and Dialectsmentioning

confidence: 99%

Automatic Language Identification in Texts: A Survey

Jauhiainen

Lui²,

Zampieri³

et al. 2019

jair

104

View full text Add to dashboard Cite

Language identification ("LI") is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used in the LI literature. We describe the features and methods using a unified notation, to make the relationships between methods clearer. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.LI as a task predates computational methods -the earliest interest in the area was motivated by the needs of translators, and simple manual methods were developed to quickly identify documents in specific languages. The earliest known work to describe a functional LI program for text is by Mustonen (1965), a statistician, who used multiple discriminant analysis to teach a computer how to distinguish, at the word level, between English, Swedish and Finnish. Mustonen compiled a list of linguistically-motivated character-based features, and trained his language identifier on 300 words for each of the three target languages. The training procedure created two discriminant functions, which were tested with 100 words for each language. The experiment resulted in 76% of the words being correctly classified; even by current standards this percentage would be seen as acceptable given the small amount of training material, although the composition of training and test data is not clear, making the experiment unreproducible.In the early 1970s, Nakamura (1971) considered the problem of automatic LI. According to Rau (1974) and the available abstract of Nakamura's article, 1 his language identifier was able to distinguish between 25 languages written with the Latin alphabet. As features, the method used the occurrence rates of characters and words in each language. From the abstract it seems that, in addition to the frequencies, he used some binary presence/absence features of particular characters or words, based on manual LI. Rau (1974) wrote his master's thesis "Language Identification by Statistical Analysis" for the Naval Postgraduate School at Monterey, California. The continued interest and the need to use LI of text in military intelligence settings is evidenced by the recent articles of, for example, Rafidha Rehiman et al. (2013), Rowe et al. (2013), and Voss et al. (2014. As features for LI, Rau (1974) used, e.g., the relative frequencies of characters and character bigrams. With a majority vote classifier ensemble of seven classifiers using Kolmogor-Smirnov's Test of Goodness of Fit and Yule's characteristic (K), he managed...

show abstract

Section: Similar Languages Language Varieties and Dialectsmentioning

confidence: 99%

Automatic Language Identification in Texts: A Survey

Jauhiainen

Lui²,

Zampieri³

et al. 2019

jair

104

View full text Add to dashboard Cite

show abstract

“…Whereas adapting large MSA/English parallel data gives significant reduction of OOV rate to 0.7% and leads to an absolute BLEU increase of 2.73 points. Salloum et al (2014) explored the impact of sentence-level dialect identification used with various linguistic features on machine translation performance. They attempted to optimize the selection of outputs produced by different MT systems given an input text including a mixture of dialects and MSA.…”

Section: Translating Between Arabic Dialects and Foreign Languagesmentioning

confidence: 99%

Machine translation for Arabic dialects (survey)

Harrat

Meftouh

Smaïli

2019

Information Processing & Management

View full text Add to dashboard Cite

Arabic dialects also called colloquial Arabic or vernaculars are spoken varieties of Standard Arabic. These dialects have mixed form with many variations due to the influence of ancient local tongues and other languages like European ones. Many of these dialects are mutually incomprehensible. Arabic dialects were not written until recently and were used only in a speech form. Nowadays, with the advent of the internet and mobile telephony technologies, these dialects are increasingly used in a written form. Indeed, this kind of communication brought everyday conversations to a written format. This allows Arab people to use their dialects, which are their actual native languages for expressing their opinion on social media, for chatting, texting, etc. This growing use opens new research direction for Arabic natural language processing (NLP). We focus, in this paper, on machine translation in the context of Arabic dialects. We provide a survey of recent research in this area. We report for each study a detailed description of the adopted approach and we give its most relevant contribution.

show abstract

“…ADAM is used in a sentence-level dialect identification approach for machine translation system selection when translating mixed dialect input (MSA and DA) (Salloum et al, 2014). We acquired two sets of training data: DA-to-English (5 M words) and MSA-to-English (57 M words).…”

Section: Dialect Identification For Mt System Selectionmentioning

confidence: 99%

ADAM: Analyzer for Dialectal Arabic Morphology

Salloum

Habash

2014

Journal of King Saud University - Computer and Information Scie

Self Cite

View full text Add to dashboard Cite

While Modern Standard Arabic (MSA) has many resources, Arabic Dialects, the primarily spoken local varieties of Arabic, are quite impoverished in this regard. In this article, we present ADAM (Analyzer for Dialectal Arabic Morphology). ADAM is a poor man's solution to quickly develop morphological analyzers for dialectal Arabic. ADAM has roughly half the outof-vocabulary rate of a state-of-the-art MSA analyzer and is comparable in its recall performance to an Egyptian dialectal morphological analyzer that took years and expensive resources to build. Ó 2014 The Authors. Production and hosting by Elsevier B.V. on behalf of King Saud University. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/3.0/).

show abstract

Sentence Level Dialect Identification for Machine Translation System Selection

Cited by 26 publications

References 17 publications

Automatic Language Identification in Texts: A Survey

Automatic Language Identification in Texts: A Survey

Machine translation for Arabic dialects (survey)

ADAM: Analyzer for Dialectal Arabic Morphology

Contact Info

Product

Resources

About