An Experimental Study of Text Preprocessing Techniques for Automatic Short Answer Grading in Indonesian

Hasanah, Uswatun; Astuti, Tri; Wahyudi, Rizki; Rifai, Zanuar; Pambudi, Rilas Agung

doi:10.1109/icitisee.2018.8720957

Cited by 23 publications

(20 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…That is important because it is a rule in language learning [23]. Data were converted into Comma-Separated Values (CSV) to facilitate further processing using the Python programming language, which has high compatibility with CSV files [24][25][26][27]. The step-by-step data pre-processing is shown in Fig.…”

Section: Methodsmentioning

confidence: 99%

Improving the accuracy of text classification using stemming method, a case of non-formal Indonesian conversation

et al. 2021

View full text Add to dashboard Cite

Background Stemming has long been used in data pre-processing to retrieve information by tracking affixed words back into their root. In an Indonesian setting, existing stemming methods have been observed, and the existing stemming methods are proven to result in high accuracy level. However, there are not many stemming methods for non-formal Indonesian text processing. This study introduces a new stemming method to solve problems in the non-formal Indonesian text data pre-processing. Furthermore, this study aims to improve the accuracy of text classifier models by strengthening stemming method. Using the Support Vector Machine algorithm, a text classifier model is developed, and its accuracy is checked. The experimental evaluation was done by testing 550 datasets in Indonesian using two different stemming methods. Findings The results show that using the proposed stemming method, the text classifier model has higher accuracy than the existing methods with a score of 0.85 and 0.73, respectively. These results indicate that the proposed stemming methods produces a classifier model with a small error rate, so it will be more accurate to predict a class of objects. Conclusion The existing Indonesian stemming methods are still oriented towards Indonesian formal sentences, therefore the method has limitations to be used in Indonesian non-formal sentences. This phenomenon underlies the suggestion of developing a corpus by normalizing Indonesian non-formal into formal to be used as a better stemming method. The impact of using the corpus as a stemming method is that it can improve the accuracy of the classifier model. In the future, the proposed corpus and stemming methods can be used for various purposes including text clustering, summarizing, detecting hate speech, and other text processing applications in Indonesian.

show abstract

Section: Methodsmentioning

confidence: 99%

Improving the accuracy of text classification using stemming method, a case of non-formal Indonesian conversation

et al. 2021

View full text Add to dashboard Cite

show abstract

“…The number of sentences which were collected in this process were 25.586. The purpose was to facilitate data processing using Python programming [17,18,19,20]. The step-by-step data pre-processing is shown in Figure. 1.…”

Section: Methodsmentioning

confidence: 99%

Improving the Accuracy of Text Classification using Stemming Method, A Case of Informal Indonesian Conversation

Rianto

Mutiara

Wibowo

et al. 2020

Preprint

View full text Add to dashboard Cite

As social beings, humans always interact with one another using either verbal or non-verbal language. Language is an arbitrary sound-symbol system, which is used by members of a community to cooperate, interact, and identify themselves. Indonesian language is classified into two categories, namely formal and non-formal. The former meets the grammatical standard as prescribed by linguistic rules of the language, while the latter tends to deviate it. In daily communication, however, non-formal language is more intensively used because they are more practical and easier to understand. With this tendency, non-formal language causes problems in linguistic computation because most linguistic computations use formal standard languages that already have standardized rules. This research aims to develop a dynamic Indonesian closed corpus related to airline ticket reservation, namely "Incorbiz". The "Incorbiz" will be used as stemming tool for formal and non-formal Indonesian. Text processing, text normalization, and auto-update data were proposed in this research. This research also compared two stemming techniques i.e. "Sastrawi" and "Incorbiz" to process the 30-sample dataset. The algorithm used to process the classification is Support Vector Machine (SVM). The data used to develop the "Incorbiz" were taken from conversations between customer service staff and consumers in airline ticket reservations. The result showed that "Incorbiz" had higher accuracy than "Sastrawi" on 0.89 and 0.67, respectively.

show abstract

“…That is important because it is a rule in language learning [23]. Data were converted into Comma-Separated Values (CSV) to facilitate further processing using the Python programming language, which has high compatibility with CSV files [24,25,26,27]. The step-by-step data pre-processing is shown in Figure. 1.…”

Section: Methodsmentioning

confidence: 99%

Improving the Accuracy of Text Classification using Stemming Method, A Case of Non-formal Indonesian Conversation

Rianto

Mutiara

Wibowo

et al. 2020

Preprint

View full text Add to dashboard Cite

Stemming has long been used in data pre-processing in information retrieval, which aims to make affix words into root words. However, there are not many stemming methods for non-formal Indonesian text processing. The existing stemming method has high accuracy for formal Indonesian, but low for non-formal Indonesian. Thus, the stemming method which has high accuracy for non-formal Indonesian classifier model is still an open-ended challenge. This study introduces a new stemming method to solve problems in the non-formal Indonesian text data pre-processing. Furthermore, this study aims to provide comprehensive research on improving the accuracy of text classifier models by strengthening on stemming method. Using the Support Vector Machine algorithm, a text classifier model is developed, and its accuracy is checked. The experimental evaluation was done by testing 550 datasets in Indonesian using two different stemming methods. The results show that using the proposed stemming method, the text classifier model has higher accuracy than the existing methods with a score of 0.85 and 0.73, respectively. In the future, the proposed stemming method can be used to develop the Indonesian text classifier model which can be used for various purposes including text clustering, summarization, detecting hate speech, and other text processing applications.

show abstract

An Experimental Study of Text Preprocessing Techniques for Automatic Short Answer Grading in Indonesian

Cited by 23 publications

References 11 publications

Improving the accuracy of text classification using stemming method, a case of non-formal Indonesian conversation

Improving the accuracy of text classification using stemming method, a case of non-formal Indonesian conversation

Improving the Accuracy of Text Classification using Stemming Method, A Case of Informal Indonesian Conversation

Improving the Accuracy of Text Classification using Stemming Method, A Case of Non-formal Indonesian Conversation

Contact Info

Product

Resources

About