Contextual Urdu Lemmatization Using Recurrent Neural Network Models

Abstract

In the field of natural language processing, machine translation is a colossally developing research area that helps humans communicate more effectively by bridging the linguistic gap. In machine translation, normalization and morphological analyses are the first and perhaps the most important modules for information retrieval (IR). To build a morphological analyzer, or to complete the normalization process, it is important to extract the correct root out of different words. Stemming and lemmatization are techniques commonly used to find the correct root words in a language. However, a few studies on IR systems for the Urdu language have shown that lemmatization is more effective than stemming due to infixes found in Urdu words. This paper presents a lemmatization algorithm based on recurrent neural network models for the Urdu language. However, lemmatization techniques for resource-scarce languages such as Urdu are not very common. The proposed model is trained and tested on two datasets, namely, the Urdu Monolingual Corpus (UMC) and the Universal Dependencies Corpus of Urdu (UDU). The datasets are lemmatized with the help of recurrent neural network models. The Word2Vec model and edit trees are used to generate semantic and syntactic embedding. Bidirectional long short-term memory (BiLSTM), bidirectional gated recurrent unit (BiGRU), bidirectional gated recurrent neural network (BiGRNN), and attention-free encoder–decoder (AFED) models are trained under defined hyperparameters. Experimental results show that the attention-free encoder-decoder model achieves an accuracy, precision, recall, and F-score of 0.96, 0.95, 0.95, and 0.95, respectively, and outperforms existing models.

A Systematic Review of Computational Approaches to Deciphering Bronze Age Aegean and Cypriot Scripts

“…(Plisson et al 2004;Stanković et al 2016;Nandathilaka, Ahangama, and Weerasuriya 2018)), machine learning-based (e.g. (Kestemont et al 2017;Freihat et al 2018;Manjavacas, Kadar, and Kestemont 2019;Akhmetov et al 2020;Karwatowski and Pietron 2022;Hafeez et al 2023)) or they employ a combination of these methods and represent hybrid models (e.g. (Ingason et al 2008;Sahala et al 2023)).…”

“…random forest classification model (Akhmetov et al 2020) and maximum entropy classifier (Freihat et al 2018)), it seems that the researchers have lately been more oriented towards the deep learning-based approaches (e.g. (Kestemont et al 2017; Manjavacas, Kadar, and Kestemont 2019; Ezhilarasi and Maheswari 2021b; Karwatowski and Pietron 2022;Hafeez et al 2023)).…”

Developing an Urdu Lemmatizer Using a Dictionary-Based Lookup Approach

“…They reported an accuracy of 86.5% for their lemmatizer. Another paper, [22], presented a lemmatization algorithm based on recurrent neural network models for the Urdu language to complete normalization and morphological processes effectively. The proposed model was trained and tested on two datasets, the Urdu Monolingual Corpus (UMC) and the Universal Dependencies Corpus of Urdu (UDU), and outperformed existing models in accuracy, precision, and recall, achieving F-scores of 0.96, 0.95, 0.95, and 0.95, respectively.…”