Evaluation of n‐gram conflation approaches for Arabic text retrieval

Abstract

In this paper we present a language-independent approach for conflation that does not depend on predefined rules or prior knowledge of the target language. The proposed unsupervised method is based on an enhancement of the pure n-gram model that can group related words based on various string-similarity measures, while restricting the search to specific locations of the target word by taking into account the order of n-grams. We show that the method is effective to achieve high score similarities for all word-form variations and reduces the ambiguity, i.e., obtains a higher precision and recall, compared to pure n-gram-based approaches for English, Portuguese, and Arabic. The proposed method is especially suited for conflation approaches in Arabic, since Arabic is a highly inflectional language. Therefore, we present in addition an adaptive user interface for Arabic text retrieval called "araSearch". araSearch serves as a metasearch interface to existing search engines. The system is able to extend a query using the proposed conflation approach such that additional results for relevant subwords can be found automatically.

Supporting Literature Review by Searching, Visualizing and Navigating Related Papers

“…Here, u and v are the words to be compared, the nested sum counts the number of n-grams in v that are similar to n-grams in a window the size of m around the same position in word v. For more details about the n-gram approach, we refer the reader to our previous work [10].…”

“…This increases the probability that the matching score between two strings can be higher even though they do not share the same concept. Therefore, we revised the computation of the similarity between words to take this aspect into account [10] and hence improve the n-gram approach precision. First of all, we check if the word is misspelled based on the dictionary entries.…”

Aara’– a system for mining the polarity of Saudi public opinion through e-newspaper comments

“…The definite article ( ال ) is always attached to nouns, and many conjunctions and prepositions are also attached as prefixes to nouns and verbs. This hinders the retrieval of morphological variants of words [23, 24]. The next example illustrates one of the challenges.…”

Talash: A Semantic and Context Based Optimized Hindi Search Engine

“…System categorized the user interest in following category and sub categories shown in table 1.Interest of a user is saved for their subsequent search [19]. Users can modify their interest for every new search.…”