Spell corrector to social media datasets in message filtering systems

Wint, Zar Zar; Ducros, Théo; Aritsugi, Masayoshi

doi:10.1109/icdim.2017.8244677

Cited by 16 publications

(6 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…A vast choice of pipelines is at our hands, we can use yandex. Speller, Damerau Levenshtein [38][39][40]. Use a version dedicated to the QA system in e-learning systems implementing the "Explicit" and "Short sentence" criteria.…”

Section: Discussionmentioning

confidence: 99%

Impact of Data Quality on Question Answering System Performances

Karra¹,

Lasfar²

2023

Intelligent Automation &Amp; Soft Computing

View full text Add to dashboard Cite

In contrast with the research of new models, little attention has been paid to the impact of low or high-quality data feeding a dialogue system. The present paper makes the first attempt to fill this gap by extending our previous work on question-answering (QA) systems by investigating the effect of misspelling on QA agents and how context changes can enhance the responses. Instead of using large language models trained on huge datasets, we propose a method that enhances the model's score by modifying only the quality and structure of the data feed to the model. It is important to identify the features that modify the agent performance because a high rate of wrong answers can make the students lose their interest in using the QA agent as an additional tool for distant learning. The results demonstrate the accuracy of the proposed context simplification exceeds 85%. These findings shed light on the importance of question data quality and context complexity construct as key dimensions of the QA system. In conclusion, the experimental results on questions and contexts showed that controlling and improving the various aspects of data quality around the QA system can significantly enhance his robustness and performance.

show abstract

Section: Discussionmentioning

confidence: 99%

Impact of Data Quality on Question Answering System Performances

Karra¹,

Lasfar²

2023

Intelligent Automation &Amp; Soft Computing

View full text Add to dashboard Cite

show abstract

“…Spelling Correction In this step, the spell checker is utilized to verify the correct words and replace the wrong words with the proper word. The 'pyspellchecker' module of Python offers the essential characteristics for checking incorrect wording [24]. Removal of stopwords Stopwords are those words in English that are added to convey the proper and intended meaning to humans, however, they do not contribute to enhancing the classification performance of machine learning models.…”

Section: Data Preprocessingmentioning

confidence: 99%

Drug Usage Safety from Drug Reviews with Hybrid Machine Learning Approach

Lee¹,

Rustam²,

Shahzad³

et al. 2023

Computer Systems Science and Engineering

View full text Add to dashboard Cite

With the increasing usage of drugs to remedy different diseases, drug safety has become crucial over the past few years. Often medicine from several companies is offered for a single disease that involves the same/similar substances with slightly different formulae. Such diversification is both helpful and dangerous as such medicine proves to be more effective or shows side effects to different patients. Despite clinical trials, side effects are reported when the medicine is used by the mass public, of which several such experiences are shared on social media platforms. A system capable of analyzing such reviews could be very helpful to assist healthcare professionals and companies for evaluating the safety of drugs after it has been marketed. Sentiment analysis of drug reviews has a large potential for providing valuable insights into these cases. Therefore, this study proposes an approach to perform analysis on the drug safety reviews using lexicon-based and deep learning techniques. A dataset acquired from the 'Drugs.Com' containing reviews of drug-related side effects and reactions, is used for experiments. A lexicon-based approach, Textblob is used to extract the positive, negative or neutral sentiment from the review text. Review classification is achieved using a novel hybrid deep learning model of convolutional neural networks and long short-term memory (CNN-LSTM) network. The CNN is used at the first level to extract the appropriate features while LSTM is used at the second level. Several well-known machine learning models including logistic regression, random forest, decision tree, and AdaBoost are evaluated using term frequency-inverse document frequency (TF-IDF), a bag of words (BoW), feature union of (TF-IDF + BoW), and lexicon-based methods. Performance analysis with machine learning models, long short term memory and convolutional neural network models, and state-of-the-art approaches indicate that the proposed CNN-LSTM model shows superior performance with an 0.96 accuracy. We also performed a statistical significance T-test to show the significance of the proposed CNN-LSTM model in comparison with other approaches.

show abstract

“…This method divides the dictionary into many clusters, and each non-word only needs to be checked in the corresponding cluster. Wint [12] added 12 methods to their spell detection and correction. These 12 methods do not use the entire dictionary as a search space, but only search the words in the dictionary that are the same as the first letter of the non-word.…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Character Embedding Correction and Candidate Word Denoising

Zheng

Lin

Jiang

2022

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

In this paper, we take Indonesian as the research object, and propose a multiple filter correction framework (MFCF). The main idea of MFCF is to remove noise from candidate words to increase the probability of correct words being selected. In MFCF, we use window search algorithm (WSA) to filter the candidate words in the dictionary. When searching for candidate words whose Levenshtein distance is 1, WSA reduces the candidate word search space by an average of 71%. When searching for candidate words whose Levenshtein distance is 2, the search space is reduced by an average of 55%. The reduction in search space has brought about an increase in search speed. When WSA searches for candidate words with Levenshtein distance equal to 1 and 2, the speed exceeds the current advanced search algorithm. A character vector-based candidate word scoring model (CWSM-CV) is also introduced in this paper. CWSM-CV is a simple but unsupervised method. In MFCF, we use CWSM-CV to filter the correct word in the candidate word list. Through exploring the feasibility of using word vector-based candidate word scoring model to score candidate words (CWSM-WV), we find the necessity of denoising the candidate word list and verified it with experiments. In order to apply this finding to the text correction, a new set of evaluation indicators are proposed to replace accuracy. Finally, we recommend that researchers who correct text in low-resource languages make the model an open system and publish it for users to use. The system receives user feedback as new data to gradually reduce the negative impact of data volume.

show abstract

Spell corrector to social media datasets in message filtering systems

Cited by 16 publications

References 6 publications

Impact of Data Quality on Question Answering System Performances

Impact of Data Quality on Question Answering System Performances

Drug Usage Safety from Drug Reviews with Hybrid Machine Learning Approach

Unsupervised Character Embedding Correction and Candidate Word Denoising

Contact Info

Product

Resources

About