UNLT: Urdu Natural Language Toolkit

Abstract

This study describes a Natural Language Processing (NLP) toolkit, as the first contribution of a larger project, for an under-resourced language—Urdu. In previous studies, standard NLP toolkits have been developed for English and many other languages. There is also a dire need for standard text processing tools and methods for Urdu, despite it being widely spoken in different parts of the world with a large amount of digital text being readily available. This study presents the first version of the UNLT (Urdu Natural Language Toolkit) which contains three key text processing tools required for an Urdu NLP pipeline; word tokenizer, sentence tokenizer, and part-of-speech (POS) tagger. The UNLT word tokenizer employs a morpheme matching algorithm coupled with a state-of-the-art stochastic n-gram language model with back-off and smoothing characteristics for the space omission problem. The space insertion problem for compound words is tackled using a dictionary look-up technique. The UNLT sentence tokenizer is a combination of various machine learning, rule-based, regular-expressions, and dictionary look-up techniques. Finally, the UNLT POS taggers are based on Hidden Markov Model and Maximum Entropy-based stochastic techniques. In addition, we have developed large gold standard training and testing data sets to improve and evaluate the performance of new techniques for Urdu word tokenization, sentence tokenization, and POS tagging. For comparison purposes, we have compared the proposed approaches with several methods. Our proposed UNLT, the training and testing data sets, and supporting resources are all free and publicly available for academic use.

Spam classification problems using support vector machine and grid search

“…-Word tokenization: word tokenization is the process of taking a piece of text and breaking it down into individual words, or tokens. In the context of email spam detection, word tokenization can be used to extract individual words from email messages to analyze and identify spam patterns (Shafi et al, 2022). By tokenizing the words in the message, we can analyze the individual words and look for certain patterns, or words, that are commonly associated with spam messages.…”

NLPashto: NLP Toolkit for Low-resource Pashto Language

“…FudanNLP employs statistics-based and rule-based methods to tackle various NLP tasks, including word segmentation, POS tagging, NER, dependency parsing, anaphora resolution, and timephrase recognition. For Urdu, a sister language of Pashto, [13] developed the UNLT toolkit, which includes three preliminary NLP tools: word tokenizer, sentence tokenizer, and partof-speech tagger. The word tokenizer utilizes a morphemematching algorithm combined with a stochastic n-gram model.…”

BaNeP: An End-to-End Neural Network Based Model for Bangla Parts-of-Speech Tagging

“…They constructed their model using CRF and ME methods for assigning the POS tag to a word [28]. Over the years, researchers such as Nunsanga MV et al [29], Shafi J et al [30], Singha RK et al [31], Dalai T et al [32] proposed probabilistic models for Mizo, Urdu, Manipuri and Odia language respectively. Transportation of a language model to another language is easier in the case of such probabilistic models.…”