Automatic identification of light stop words for Persian information retrieval systems

Abstract

Stop word identification is one of the most important tasks for many text processing applications such as information retrieval. Stop words occur too frequently in documents in a collection and do not contribute significantly to determining the context or information about the documents. These words are worthless as index terms and should be removed during indexing as well as before querying by an information retrieval system. In this paper, we propose an automatic aggregated methodology based on term frequency, normalized inverse document frequency and information model to extract the light stop words from Persian text. We define a ‘light stop word’ as a stop word that has few letters and is not a compound word. In the Persian language, a complete stop word list can be derived by combining the light stop words. The evaluation results, using a standard corpus, show a good percentage of coincidence between the Persian and English stop words and a significant improvement in the number of index terms. Specifically, the first 32 Persian light stop words have a great impact on the index size reduction and the set of stop words can reduce the number of index terms by about 27%.

Integrating word status for joint detection of sentiment and aspect in reviews

“…By assessing the results for EElectronics and PElectronics, it can be found that the results for EElectronics are slightly better. Using a complex script is the main challenge in Persian writing [21,51,52]. For example, one of the issues in Persian text mining is the wide variety of declensional suffixes.…”

Proceedings of Workshop for NLP Open Source Software (NLP-OSS)

“…Stop lists have been generated for other languages, such as Chinese (Zou et al, 2006), Thai (Daowadung and Chen, 2012) and Farsi (Sadeghi and Vegas, 2014), using uses similar frequency threshold approaches, are susceptible to the same issues discussed here.…”

Construction and evaluation of PHP‐based management and training system for electrical power laboratory

“…In order to better improve and use the system, we add some auxiliary equipment and materials to the online management and training environment, where efficient intelligence supporting is one of the main characteristics. There are already many systems supporting various types of intelligence, including attendance management , automatic identification ID card information , and automatic mark examination papers . However, to separate the various functions will affect the management efficiency, and increasing the costs of learning.…”

“…There are already many systems Correspondence to H. Xu (railway_dragon@163.com). supporting various types of intelligence, including attendance management [16], automatic identification ID card information [17], and automatic mark examination papers [18]. However, to separate the various functions will affect the management efficiency, and increasing the costs of learning.…”