Improved TFIDF weighting techniques in document Retrieval

Yamout, Fadi; Lakkis, Rachad

doi:10.1109/icdim.2018.8847156

Cited by 9 publications

(2 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Once the Document Frequency value of each feature is calculated, appropriate features are selected through the threshold. (Kshirsagar et al, 2020;Mestry et al, 2019;Gao et al, 2019;Kermani et al, 2019;Othman et al, 2019;Sidorov et al, 2019;Maryam et al, 2018;Das et al, 2018;Yamout et al, 2018;White et al, 2018;Alsmadi et al, 2018;Gu et al, 2018;Emelyanov et al, 2017;Chen et al, 2017;Krouska et al, 2016).…”

Section: Features Selection and Extractionmentioning

confidence: 99%

Influence of Pre-Processing Strategies on the Performance of ML Classifiers Exploiting TF-IDF and BOW Features

Pimpalkar

Raj

2020

ADCAIJ

View full text Add to dashboard Cite

Data analytics and its associated applications have recently become impor-tant fields of study. The subject of concern for researchers now-a-days is a massive amount of data produced every minute and second as people con-stantly sharing thoughts, opinions about things that are associated with them. Social media info, however, is still unstructured, disseminated and hard to handle and need to be developed a strong foundation so that they can be utilized as valuable information on a particular topic. Processing such unstructured data in this area in terms of noise, co-relevance, emoticons, folksonomies and slangs is really quite challenging and therefore requires proper data pre-processing before getting the right sentiments. The dataset is extracted from Kaggle and Twitter, pre-processing performed using NLTK and Scikit-learn and features selection and extraction is done for Bag of Words (BOW), Term Frequency (TF) and Inverse Document Frequency (IDF) scheme. For polarity identification, we evaluated five different Machine Learning (ML) algorithms viz Multinomial Naive Bayes (MNB), Logistic Regression (LR), Decision Trees (DT), XGBoost (XGB) and Support Vector Machines (SVM). We have performed a comparative analysis of the success for these algorithms in order to decide which algorithm works best for the given data-set in terms of recall, accuracy, F1-score and precision. We assess the effects of various pre-processing techniques on two datasets; one with domain and other not. It is demonstrated that SVM classifier outperformed the other classifiers with superior evaluations of 73.12% and 94.91% for accuracy and precision respectively. It is also highlighted in this research that the selection and representation of features along with various pre-processing techniques have a positive impact on the performance of the classification. The ultimate outcome indicates an improvement in sentiment classification and we noted that pre-processing approaches obviously suggest an improvement in the efficiency of the classifiers.

show abstract

Section: Features Selection and Extractionmentioning

confidence: 99%

Influence of Pre-Processing Strategies on the Performance of ML Classifiers Exploiting TF-IDF and BOW Features

Pimpalkar

Raj

2020

ADCAIJ

View full text Add to dashboard Cite

show abstract

“…TF-IDF is included in the process of counting the number of words. The word appears in each document [10], [11]. Equation 1 is no TF-IDF formula.…”

Section: Tf-idfmentioning

confidence: 99%

Application of Density Based Clustering of Disaster Location in Realtime Social Media

Widianto¹,

Sudirman²,

Awaluddin³

2020

TEM Journal

View full text Add to dashboard Cite

Online life is used as a method of finding information, one of which is Twitter as the medium. The occurrence of natural disasters is very detrimental. Therefore, the application is needed to see natural disasters through social media Twitter. A small number of studies using clustering methods based on Twitter user data density are the beginning of this research. With the availability of data in certain areas makes it easy to group. After that, the data is grouped based on a high degree of similarity. One result of applying this method is the location of the disaster. NER-based rules are used to discover out the area of the disaster. Data accuracy testing is performed using the Silhouette coefficient.

show abstract

Fast and Memory-Efficient TFIDF Calculation for Text Analysis of Large Datasets

Senbel

2021

Advances and Trends in Artificial Intelligence. Artificial Intelligence Practices

View full text Add to dashboard Cite

Improved TFIDF weighting techniques in document Retrieval

Cited by 9 publications

References 4 publications

Influence of Pre-Processing Strategies on the Performance of ML Classifiers Exploiting TF-IDF and BOW Features

Influence of Pre-Processing Strategies on the Performance of ML Classifiers Exploiting TF-IDF and BOW Features

Application of Density Based Clustering of Disaster Location in Realtime Social Media

Fast and Memory-Efficient TFIDF Calculation for Text Analysis of Large Datasets

Contact Info

Product

Resources

About