The Enhancement of Arabic Stemming by Using Light Stemming and Dictionary-Based Stemming

Alhanini, Yasir; Aziz, Mohd Juzaiddin Ab

doi:10.4236/jsea.2011.49060

Cited by 9 publications

(5 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, this method cannot model non-contiguous roots, of which Semitic languages have many. Other unsupervised methods utilize dictionaries to select the characters from within words (Darwish, 2002;Boudlal et al, 2011;Alhanini and Ab Aziz, 2011). Another line of research leverages the templatic nature for human-constructed rule-based constraints (Elghamry, 2005;Rodrigues and Cavar, 2007;Choueka, 1990).…”

Section: Related Workmentioning

confidence: 99%

Constrained Sequence-to-sequence Semitic Root Extraction for Enriching Word Embeddings

El-Kishky

Fu²,

Addawood

et al. 2019

Proceedings of the Fourth Arabic Natural Language Processing Workshop

View full text Add to dashboard Cite

In this paper, we tackle the problem of "root extraction" from words in the Semitic language family. A challenge in applying natural language processing techniques to these languages is the data sparsity problem that arises from their rich internal morphology, where the substructure is inherently non-concatenative and morphemes are interdigitated in word formation. While previous automated methods have relied on human-curated rules or multiclass classification, they have not fully leveraged the various combinations of regular, sequential concatenative morphology within the words and the internal interleaving within templatic stems of roots and patterns. To address this, we propose a constrained sequence-tosequence root extraction method. Experimental results show our constrained model outperforms a variety of methods at root extraction. Furthermore, by enriching word embeddings with resulting decompositions, we show improved results on word analogy, word similarity, and language modeling tasks.

show abstract

Section: Related Workmentioning

confidence: 99%

Constrained Sequence-to-sequence Semitic Root Extraction for Enriching Word Embeddings

El-Kishky

Fu²,

Addawood

et al. 2019

Proceedings of the Fourth Arabic Natural Language Processing Workshop

View full text Add to dashboard Cite

show abstract

“…approaches for root-andpattern morphology involve rule-based root extraction (Khoja and Garside, 1999;Taghva et al, 2005;Ababneh et al, 2012;El-Beltagy and Rafea, 2011;Alhanini et al, 2011;Al-Shalabi and Evens, 1998), and others that require training data of word-root pairs (Attia et al, 2016;Al-Serhan and Ayesh, 2006), all developed for processing Arabic. The resource-intensive nature of these methods not only limits their wider applicability to other Semitic languages but also prevents from handling the productive process more generally.…”

Section: Currentmentioning

confidence: 99%

Fixing the Infix: Unsupervised Discovery of Root-and-Pattern Morphology

Sakakini,

Bhat,

Viswanath

2017

Preprint

View full text Add to dashboard Cite

We present an unsupervised and languageagnostic method for learning root-andpattern morphology in Semitic languages. We harness the syntactico-semantic information in distributed word representations to solve the long standing problem of root-and-pattern discovery in Semitic languages. The root-and-pattern morphological rules we learn in an unsupervised manner are validated by native speakers in Arabic, Hebrew, and Amharic. Further, the rules are used to construct an unsupervised root extractor called JZR, which we show to compare favorably with ISRI, a widely used and carefully engineered root extractor for Arabic.

show abstract

“…The study found the hybrid approach to have an average accuracy of approximately 89.3%. A rule-based stemmer for Bengali language which uses stem dictionary for further validation was developed in [8], whereas an enhanced stemmer for Arabic was built by integrating light and dictionary based stemming [9]. The latter study found their stemmer to produce an average accuracy of approximately 96%.…”

Section: Stemmingmentioning

confidence: 99%

Improving Document Relevancy Using Integrated Language Modeling Techniques

Balakrishnan

Humaidi

Lloyd-Yemoh

2016

MJCS

View full text Add to dashboard Cite

This paper presents an integrated language model to improve document relevancy for text-queries. To be precise, an integrated stemming-lemmatization (S-L) model was developed and its retrieval performance was compared at three document levels, that is, at top 5, 10 and 15. A prototype search engine was developed and fifteen queries were executed. The mean average precisions revealed the S-L model to outperform the baseline (i.e. no language processing), stemming and also the lemmatization models at all three levels of the documents. These results were also supported by the histogram precisions which illustrated the integrated model to improve the document relevancy. However, it is to note that the precision differences between the various models were insignificant. Overall the study found that when language processing techniques, that is, stemming and lemmatization are combined, more relevant documents are retrieved.Keywords: Information retrieval, document relevancy, language modeling, stemming, lemmatization, mean average precision INTRODUCTIONThe use of internet all over the world has caused information size to increase, hence making it possible for large volumes of information to be retrieved by the users. However, this phenomenon also makes it difficult for users to find relevant information, therefore proper information retrieval techniques are needed. Information retrieval can be defined as "a problem-oriented discipline concerned with the problem of the effective and efficient transfer of desired information between human generator and human user" [1]. In short, information retrieval aims to provide users with those documents that will satisfy their information need.Many information retrieval algorithms were proposed, and some of the popular ones include the traditional Boolean model (i.e. based on binary decisions), vector space model (i.e. compares user queries with documents found in collections and computes their similarities), and probabilistic model (i.e. based on the probability theory to model uncertainties involved in retrieving data), among others. Over the years, information retrieval has evolved to include text retrieval in different languages, and thus giving birth to language models. The language model is particularly concerned with identifying how likely it is for a particular string in a specific language to be repeated [2]. A popular technique used in the language model is the N-gram model which predicts a preceding word based on previous N-1 words [3]. Other popular techniques include stemming and lemmatization.

show abstract

The Enhancement of Arabic Stemming by Using Light Stemming and Dictionary-Based Stemming

Cited by 9 publications

References 9 publications

Constrained Sequence-to-sequence Semitic Root Extraction for Enriching Word Embeddings

Constrained Sequence-to-sequence Semitic Root Extraction for Enriching Word Embeddings

Fixing the Infix: Unsupervised Discovery of Root-and-Pattern Morphology

Improving Document Relevancy Using Integrated Language Modeling Techniques

Contact Info

Product

Resources

About