Dynamic Stopword Removal for Sinhala Language

Jayaweera, Akash; Senanayake, Y.N; Haddela, Prasanna S.

doi:10.1109/nitc48475.2019.9114476

Cited by 5 publications

(5 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Filtering text has rules for removing noise data and using the stopword removal method. Stopword removal is a step to filter out words that contain little information and keep meaningful words [10]. In this work, we used a slang word dictionary from colloquial-indonesian-lexicon to be filtered slang words and used a stopword dictionary from dataset stopwords-id-satya.…”

Section: B Natural Language Processingmentioning

confidence: 99%

Sentiment Analysis of Indonesian TikTok Review Using LSTM and IndoBERTweet Algorithm

Setiawan,

M. Lhaksmana,

Bunyamin

2023

jipi. jurnal. ilmiah. penelitian. dan. pembelajaran. informatik

View full text Add to dashboard Cite

TikTok is currently the most popular app in the world and thus gets many reviews on the Google Play Store and other app marketplace platforms. These reviews are valuable user opinions that can be analyzed further for many purposes. Harnessing valuable analyses from these reviews can be obtained manually, which will be time-consuming and costly, or automatically with machine learning methods. This paper implements the latter with LSTM and IndoBERTweet, a derivative of BERT, using Indonesian vocabulary from Twitter post data. This research aims to determine the appropriate method to create a model that can automatically classify TikTok reviews into negative, neutral, and positive sentiments. The result demonstrates that IndoBERTweet outperforms the other, with an accuracy of 80%, whereas the LSTM accuracy is at 78%.

show abstract

Section: B Natural Language Processingmentioning

confidence: 99%

Sentiment Analysis of Indonesian TikTok Review Using LSTM and IndoBERTweet Algorithm

Setiawan,

M. Lhaksmana,

Bunyamin

2023

jipi. jurnal. ilmiah. penelitian. dan. pembelajaran. informatik

View full text Add to dashboard Cite

show abstract

“…Jayaweera et al [15] proposed a dynamic approach to find Sinhala stopwords. In this study, they argued the cutoff point is subjective to the dataset.…”

Section: Related Workmentioning

confidence: 99%

“…In this study, term frequency (TF), inverse document frequency (IDF), and term-frequency-inverse-documentfrequency (TF-IDF) are calculated for every term in the dataset. Authors select this approach since it is independent of the dataset size and domain, and it has been used in many other low-resource languages [15]. Further authors conducted multiple executions with different values to define the threshold.…”

Section: B Stopword Identificationmentioning

confidence: 99%

Towards Stopwords Identification in Tamil Text Clustering

Fayaza¹,

Farhath²

2021

IJACSA

View full text Add to dashboard Cite

Now-a-days, digital documents have become the primary source of information. Therefore, natural language processing is widely utilized in information retrieval, topic modeling, document classification, and document clustering. Preprocessing plays a significant role in all of these applications. One of the critical steps in preprocessing is removing stopwords. Many languages have defined their list of stopwords. However, a publicly available stopwords list isn't available for the Tamil language since it is under-resourced. This study identified 93 general and some domain-specific stopwords for sports, entertainment, local and foreign news by analyzing more than 1.7 million Tamil documents with more than 21 million words. Also, this study shows that removing stopwords improves the accuracy of a Tamil document clustering system. It showed an improvement of 2.4%, 0.95% in the F-score for TF-IDF with one pass algorithm and FastText with the one-pass algorithm, respectively.

show abstract

“…Our stopword list was produced using tf-idf ranking and consists of 210 Sinhala words. We further tested it using a number of classifierssee [11] for details. This list is called inside the Lucene Sinhala analyzer that we have developed.…”

Section: Sinhala Stopword Listmentioning

confidence: 99%

Use of Interpretable Evolved Search Query Classifiers for Sinhala Documents

Haddela

Hirsch

Brunsdon

et al. 2020

Proceedings of the Future Technologies Conference (FTC) 2020, Volume 1

View full text Add to dashboard Cite

Document analysis is a well matured yet still active research field, partly as a result of the intricate nature of building computational tools but also due to the inherent problems arising from the variety and complexity of human languages. Breaking down language barriers is vital in enabling access to a number of recent technologies. This paper investigates the application of document classification methods to new Sinhalese datasets. This language is geographically isolated and rich with many of its own unique features. We will examine the interpretability of the classification models with a particular focus on the use of evolved Lucene search queries generated using a Genetic Algorithm (GA) as a method of document classification. We will compare the accuracy and interpretability of these search queries with other popular classifiers. The results are promising and are roughly in line with previous work on English language datasets.

show abstract

Dynamic Stopword Removal for Sinhala Language

Cited by 5 publications

References 13 publications

Sentiment Analysis of Indonesian TikTok Review Using LSTM and IndoBERTweet Algorithm

Sentiment Analysis of Indonesian TikTok Review Using LSTM and IndoBERTweet Algorithm

Towards Stopwords Identification in Tamil Text Clustering

Use of Interpretable Evolved Search Query Classifiers for Sinhala Documents

Contact Info

Product

Resources

About