On Continent and Script-Wise Divisions-Based Statistical Measures for Stop-words Lists of International Languages

Saini, Jatinderkumar R.; Rakholia, Rajnish M.

doi:10.1016/j.procs.2016.06.076

Cited by 26 publications

(16 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We would like to emphasize here that the researchers lemmatized the words after identifying a word as a stop word as opposed to lemmatizing all the words and then identifying the stop words. [17] collated a list of stop words in numerous languages spanning multiple countries. www.ijacsa.thesai.org…”

Section: Existing Work and Objectivesmentioning

confidence: 99%

Novel Language Resources for Hindi: An Aesthetics Text Corpus and a Comprehensive Stop Lemma List

Venugopal-Wairagade¹,

Saini²,

Pramod³

2020

IJACSA

View full text Add to dashboard Cite

This paper is an effort to complement the contributions made by researchers working toward the inclusion of non-English languages in natural language processing studies. Two novel Hindi language resources have been created and released for public consumption. The first resource is a corpus consisting of nearly thousand pre-processed fictional and nonfictional texts spanning over hundred years. The second resource is an exhaustive list of stop lemmas created from 12 corpora across multiple domains, consisting of over 13 million words, from which more than 200,000 lemmas were generated, and 11 publicly available stop word lists comprising over 1000 words, from which nearly 400 unique lemmas were generated. This research lays emphasis on the use of stop lemmas instead of stop words owing to the presence of various, but not all morphological forms of a word in stop word lists, as opposed to the presence of only the root form of the word, from which variations could be derived if required. It was also observed that stop lemmas were more consistent across multiple sources as compared to stop words. In order to generate a stop lemma list, the parts of speech of the lemmas were investigated but rejected as it was found that there was no significant correlation between the rank of a word in the frequency list and its part of speech. The stop lemma list was assessed using a comparative method. A formal evaluation method is suggested as future work arising from this study.

show abstract

Section: Existing Work and Objectivesmentioning

confidence: 99%

Novel Language Resources for Hindi: An Aesthetics Text Corpus and a Comprehensive Stop Lemma List

Venugopal-Wairagade¹,

Saini²,

Pramod³

2020

IJACSA

View full text Add to dashboard Cite

show abstract

“…In the case of the English language, sets of stop-words widely used in Natural Language Processing are used in text cleaning tasks. This work uses a the publicly-available English stop-words set published in [33], and each word is weighted by textual and lexical functions in a sentence [34]. URL patterns are removed from the corpus and other expressions, such as retweets RT and appearances of @username, are considered non-informative attributes and are deleted in the same way.…”

Section: Tokenization and Noise Removalmentioning

confidence: 99%

Social Sentiment Sensor in Twitter for Predicting Cyber-Attacks Using ℓ1 Regularization

Hernandez-Suarez¹,

Sánchez-Pérez²,

Toscano-Medina³

et al. 2018

Preprint

View full text Add to dashboard Cite

Abstract:In recent years, online social media information has been subject of study in several data science fields due to its impact on users as a communication and expression channel. Data gathered from online platforms such as Twitter has the potential to facilitate research over social phenomena based on sentiment analysis, which usually employs Natural Language Processing and Machine Learning techniques to interpret sentimental tendencies related to users opinions and make predictions about real events. Cyber attacks are not isolated from opinion subjectivity on online social networks. Various security attacks are performed by hacker activists motivated by reactions from polemic social events. In this paper, a methodology for tracking social data that can trigger cyber attacks is developed. Our main contribution lies in the monthly prediction of tweets with content related to security attacks and the incidents detected based on 1 regularization.

show abstract

“…The algorithm also tested on 200 documents and succeeded 99% accuracy and time efficiency. Saini and Rakholia [13] have presented an analytic in-depth report on continent and script-wise divisions-based statistical measures for stopwords lists of various international Languages. A. Alajmi et al [19] generated stop-words for the Arabic language using a statistical approach.1002 documents with over 700,000 words were tested and they achieved about 90% general accuracy.…”

Section: Related Workmentioning

confidence: 99%

Automatic Generation of Stopwords in the Amharic Text

Miretie¹,

Khedkar²

2018

IJCA

View full text Add to dashboard Cite

For the retrieval of information from documents of different natural languages, pre-processing of the document is the main task. During pre-processing, words which occur too frequently and have little semantic in the document should be identified. Such words are called Stopwords. Stopwords list for different world languages like English, Chinese, Hindi, Arabic Sanskrit etc. are identified. But as I long as I know there is no standard method to identify these words for the Amharic language. In this paper, we proposed the automatic identification of Stopwords for the Amharic text by an aggregate based methodology of words frequency, inverse document frequency, and entropy value measure. Available works on Stopwords identification techniques are based on static or dictionary based Stopwords lists. This method inefficient and very expensive and it is a time-consuming task as the searching process takes a long time. The proposed work will overcome these problems using aggregated methods of both frequency measures and entropy measures of words in the Amharic text for the automatic Stopwords identification.

show abstract

On Continent and Script-Wise Divisions-Based Statistical Measures for Stop-words Lists of International Languages

Cited by 26 publications

References 2 publications

Novel Language Resources for Hindi: An Aesthetics Text Corpus and a Comprehensive Stop Lemma List

Novel Language Resources for Hindi: An Aesthetics Text Corpus and a Comprehensive Stop Lemma List

Social Sentiment Sensor in Twitter for Predicting Cyber-Attacks Using ℓ1 Regularization

Automatic Generation of Stopwords in the Amharic Text

Contact Info

Product

Resources

About