Enhancement of Text Analysis Using Context-Aware Normalization of Social Media Informal Text

Khan, Jebran; Lee, Sungchang

doi:10.3390/app11178172

Cited by 6 publications

(8 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Table 5 lists the number of correct, incorrect, and non-normalized words. These same 300 words were then put through the normalization methods proposed by [ 12 , 39 ]. Both these models were able to use the regular expression method and a spell-check algorithm to normalize OOV words with repeated letters resulting in impressive outcomes.…”

Section: Experimental Evaluation and Resultsmentioning

confidence: 99%

“…Both these models were able to use the regular expression method and a spell-check algorithm to normalize OOV words with repeated letters resulting in impressive outcomes. Table 5 provides a comparison of the outcomes of the RBPsWRL- Sym model and that of the normalization models proposed by [ 12 , 39 ]. As seen in Fig 20 , the RBPsWRL-Sym model increased the F1 score from 78% and 81% to 88%.…”

Section: Experimental Evaluation and Resultsmentioning

confidence: 99%

“…As seen in Fig 20 , the RBPsWRL-Sym model increased the F1 score from 78% and 81% to 88%. Therefore, according to the F1 scores, the RBPsWRL-Sym model performed 9% better than the normalization model by [ 12 ] and 13% better than that of [ 39 ] (Figs 20 and 21 ). Both these methods truncate repeated letters followed by a spelling correction algorithm.…”

Section: Experimental Evaluation and Resultsmentioning

confidence: 99%

“…The TRL1 algorithm is utilized to truncate the letters to one instance. It is uncommon for letters to be repeated more than once in English [ 39 ]. It is even rarer for letters to be repeated at the beginning of an English word.…”

Section: Methodsmentioning

confidence: 99%

See 3 more Smart Citations

A normalization model for repeated letters in social media hate speech text based on rules and spelling correction

Mansur,

Omar,

Tiun

et al. 2024

PLoS ONE

View full text Add to dashboard Cite

As social media booms, abusive online practices such as hate speech have unfortunately increased as well. As letters are often repeated in words used to construct social media messages, these types of words should be eliminated or reduced in number to enhance the efficacy of hate speech detection. Although multiple models have attempted to normalize out-of-vocabulary (OOV) words with repeated letters, they often fail to determine whether the in-vocabulary (IV) replacement words are correct or incorrect. Therefore, this study developed an improved model for normalizing OOV words with repeated letters by replacing them with correct in-vocabulary (IV) replacement words. The improved normalization model is an unsupervised method that does not require the use of a special dictionary or annotated data. It combines rule-based patterns of words with repeated letters and the SymSpell spelling correction algorithm to remove repeated letters within the words by multiple rules regarding the position of repeated letters in a word, be it at the beginning, middle, or end of the word and the repetition pattern. Two hate speech datasets were then used to assess performance. The proposed normalization model was able to decrease the percentage of OOV words to 8%. Its F1 score was also 9% and 13% higher than the models proposed by two extant studies. Therefore, the proposed normalization model performed better than the benchmark studies in replacing OOV words with the correct IV replacement and improved the performance of the detection model. As such, suitable rule-based patterns can be combined with spelling correction to develop a text normalization model to correctly replace words with repeated letters, which would, in turn, improve hate speech detection in texts.

show abstract

Section: Experimental Evaluation and Resultsmentioning

confidence: 99%

Section: Experimental Evaluation and Resultsmentioning

confidence: 99%

Section: Experimental Evaluation and Resultsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

See 2 more Smart Citations

A normalization model for repeated letters in social media hate speech text based on rules and spelling correction

Mansur,

Omar,

Tiun

et al. 2024

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…This approach shows good performance for text normalization for Indonesian TTS with a Word Error Rate (WER) of 0.0805. Another study conducted by (Khan & Lee, 2021) concluded that In this research, it is proposed to develop an application called Textual Variations Handler (TVH), which is a generic application that works in a variation-independent manner to handle various types of noise in textual data originating from various social media (SM) applications to improve text analysis. The aim of this research is to introduce a hybrid normalization technique that is effective in ensuring that information obtained from noisy text data can be utilized in the desired form.…”

Section: Literature Reviewmentioning

confidence: 99%

Twitter Data Analysis and Text Normalization in Collecting Standard Word

Lubis

Nasution

2023

JAETS

View full text Add to dashboard Cite

is one of the most important data sources in social data analysis. However, the text contained on Twitter is often unstructured, resulting in difficulties in collecting standard words. Therefore, in this research, we analyze Twitter data and normalize text to produce standard words that can be used in social data analysis. The purpose of this research is to improve the quality of data collection on standard words on social media from Twitter and facilitate the analysis of social data that is more accurate and valid. The method used is natural language processing techniques using classification algorithms and text normalization techniques. The result of this study is a set of standard words that can be used for social data analysis with a total of 11430 words, then 4075 words with structural or formal words and 7355 informal words. Informal words are corrected by trusted sources to create a corpus of formal and informal words obtained from social media tweet data @fullSenyum. The contribution to this research is that the method developed can improve the quality of social data collection from Twitter by ensuring the words used are standard and accurate and the text normalization method used in this study can be used as a reference for text normalization in other social data, thus facilitating collection. and better-quality social data analysis. This research can assist researchers or practitioners in understanding natural language processing techniques and their application in social data analysis. This research is expected to assist in collecting social data more effectively and efficiently.

show abstract

A human-machine interaction framework for identifying factors influential consumer participation in e-waste treatment schemes

Nowruzi

Shokouhyar

Dehghan

et al. 2023

International Journal of Computer Integrated Manufacturing

View full text Add to dashboard Cite

Enhancement of Text Analysis Using Context-Aware Normalization of Social Media Informal Text

Cited by 6 publications

References 43 publications

A normalization model for repeated letters in social media hate speech text based on rules and spelling correction

A normalization model for repeated letters in social media hate speech text based on rules and spelling correction

Twitter Data Analysis and Text Normalization in Collecting Standard Word

A human-machine interaction framework for identifying factors influential consumer participation in e-waste treatment schemes

Contact Info

Product

Resources

About