English to Urdu Statistical Machine Translation: Establishing a Baseline

Jawaid, Bushra; Kamran, Amir; Bojar, Ondřej

doi:10.3115/v1/w14-5505

Cited by 14 publications

(14 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To create our proposed dictionary, we collected data from two different sources: (1) the Urdu Mono-lingual (UrMono) Corpus (https://lindat.mff.cuni.cz/repository/xmlui/handle/ 11858/00-097C-0000-0023-65A9-5, which was last visited on 14 March 2023) by [23] and (2) the Urdu Wikipedia dump (http://wikipedia.c3sl.ufpr.br/urwiki/20191120/, accessed on 14 March 2023). In the first step, the source text from the UrMono corpus was pre-processed (we identified and removed stop words, URLs, digits, and English alphabets) and tokenized (tokenization was done with the help of the Urdu word tokenizer developed by [24]).…”

Section: Data Sourcementioning

confidence: 99%

Developing an Urdu Lemmatizer Using a Dictionary-Based Lookup Approach

2023

View full text Add to dashboard Cite

Lemmatization aims at returning the root form of a word. The lemmatizer is envisioned as a vital instrument that can assist in many Natural Language Processing (NLP) tasks. These tasks include Information Retrieval, Word Sense Disambiguation, Machine Translation, Text Reuse, and Plagiarism Detection. Previous studies in the literature have focused on developing lemmatizers using rule-based approaches for English and other highly-resourced languages. However, there have been no thorough efforts for the development of a lemmatizer for most South Asian languages, specifically Urdu. Urdu is a morphologically rich language with many inflectional and derivational forms. This makes the development of an efficient Urdu lemmatizer a challenging task. A standardized lemmatizer would contribute towards establishing much-needed methodological resources for this low-resourced language, which are required to boost the performance of many Urdu NLP applications. This paper presents a lemmatization system for the Urdu language, based on a novel dictionary lookup approach. The contributions made through this research are the following: (1) the development of a large benchmark corpus for the Urdu language, (2) the exploration of the relationship between parts of speech tags and the lemmatizer, and (3) the development of standard approaches for an Urdu lemmatizer. Furthermore, we experimented with the impact of Part of Speech (PoS) on our proposed dictionary lookup approach. The empirical results showed that we achieved the best accuracy score of 76.44% through the proposed dictionary lookup approach.

show abstract

Section: Data Sourcementioning

confidence: 99%

Developing an Urdu Lemmatizer Using a Dictionary-Based Lookup Approach

2023

View full text Add to dashboard Cite

show abstract

“…Complex words dictionary: To address the space insertion problem, a large complex words dictionary was created using the UMC Urdu data set (Jawaid, Kamran, and Bojar 2014), which contains data from various domains including Sports, Politics, Blogs, Education, Literature, Entertainment, Science, Technology, Commerce, Health, Law, Business, Showbiz, Fiction, and Weather. From each domain, at least 1000 sentences were randomly selected and preprocessed to remove noise (see Section 5.4).…”

Section: Urdu Word Tokenizer 411 Generating Supporting Resources For ...mentioning

confidence: 99%

UNLT: Urdu Natural Language Toolkit

et al. 2022

View full text Add to dashboard Cite

This study describes a Natural Language Processing (NLP) toolkit, as the first contribution of a larger project, for an under-resourced language—Urdu. In previous studies, standard NLP toolkits have been developed for English and many other languages. There is also a dire need for standard text processing tools and methods for Urdu, despite it being widely spoken in different parts of the world with a large amount of digital text being readily available. This study presents the first version of the UNLT (Urdu Natural Language Toolkit) which contains three key text processing tools required for an Urdu NLP pipeline; word tokenizer, sentence tokenizer, and part-of-speech (POS) tagger. The UNLT word tokenizer employs a morpheme matching algorithm coupled with a state-of-the-art stochastic n-gram language model with back-off and smoothing characteristics for the space omission problem. The space insertion problem for compound words is tackled using a dictionary look-up technique. The UNLT sentence tokenizer is a combination of various machine learning, rule-based, regular-expressions, and dictionary look-up techniques. Finally, the UNLT POS taggers are based on Hidden Markov Model and Maximum Entropy-based stochastic techniques. In addition, we have developed large gold standard training and testing data sets to improve and evaluate the performance of new techniques for Urdu word tokenization, sentence tokenization, and POS tagging. For comparison purposes, we have compared the proposed approaches with several methods. Our proposed UNLT, the training and testing data sets, and supporting resources are all free and publicly available for academic use.

show abstract

“…The TBL approaches derives rules automatically from the corpus and produces more accuracy as compared to other approaches and provides more advantages than any other approach used for tagging systems. In [23], the new Urdu POS tag set design schema is presented. The overall system accuracy is reported as 96.8%.…”

Section: Urdu Pos Taggingmentioning

confidence: 99%

Analysis and Comparative Study of POS Tagging Techniques for National (Urdu) Language and other Regional Languages of Pakistan

Rajper¹,

Rajper²,

Maitlo³

et al. 2021

Sindh Univ. Res. Jour

View full text Add to dashboard Cite

Defining algorithms and techniques to enable computers to understand human language is the Natural Language Processing (NLP), which is an integral part of speech recognition. Parts of Speech (POS) is considered as one of the well understood problems of Natural Language Processing, in which natural language words and sentence are tagged or assigned grammatical classes, because tagging a single word by human hand is a time consuming and tedious job. To automate the tagging job is the way to automate the lexicons of the text of a language. Many of the languages are enriched with their POS tagging systems. Pakistani regional languages are less developed due to the many reasons and much of the work is needed in POS tagging system. Some of the regional languages have their POS tagging systems but still they need some more attention to refine their system. Some of the languages need to develop from the scratch. Balochi language has no any POS tagging system. This study presents the comparative analysis of POS tagging approaches for the national language (Urdu) and other regional languages of Pakistan. The approaches and their data sets used and their reported results are presented here

show abstract

English to Urdu Statistical Machine Translation: Establishing a Baseline

Cited by 14 publications

References 11 publications

Developing an Urdu Lemmatizer Using a Dictionary-Based Lookup Approach

Developing an Urdu Lemmatizer Using a Dictionary-Based Lookup Approach

UNLT: Urdu Natural Language Toolkit

Analysis and Comparative Study of POS Tagging Techniques for National (Urdu) Language and other Regional Languages of Pakistan

Contact Info

Product

Resources

About