Effects of Stratified Cross-Validation and Hyperparameter Tuning on Sentiment Classification With the Chi2-Rfe Hybrid Feature Selection Technique in the Imdb Dataset

Abstract

Data analysis from social networking sites provides government entities, businesses, and event planners with insights into public sentiments and perceptions. Sentiment analysis (SA) resolves this need by classifying the sentiment of social network users into multiple classes. Despite their usefulness, data from social networking platforms frequently exhibits challenges, including unstructured formats, high volume, and redundant or irrelevant information, which can cause issues like overfitting, underfitting, and the curse of dimensionality. In response to these challenges, this study proposes using the term frequency-inverse document frequency (TF-IDF) for feature extraction along with a hybrid feature selection method that combines Chi2 and recursive feature elimination (RFE), called Chi2-RFE. This approach seeks to identify the optimal feature subset by filtering out irrelevant and redundant features. The proposed method is tested with several classifiers, including KNN, LR, SVC, GNB, DT, and RFC, employing stratified K-fold cross-validation and hyperparameter tuning on an IMDb dataset obtained from Kaggle. By effectively addressing overfitting and underfitting issues, this approach shows that before using StratefiedKfold cross-validation and hyperparameter tuning, LR gives 0.81975 training accuracy and test accuracy 0.815 on training data. After the method mentioned above, overfitting is removed by enhancing accuracy to 0.864833 on test data. KNN also enhanced its test accuracy to 0.891667 from 0.857333. SVC from 0.846666 to 0.883667, and GNB from 0.809666 to 0.829583. Precision is also improved from 0.826 to 0.853 for LR, from 0.848 to 0.897 for KNN, from 0.852 to 0.868 for SVC, and from 0.809666 to 0.799 for GNB. Recall also shows improvement from 0.815 to 0.600 for LR, from 0.857 to 0.894 for KNN, from 0.847 to 0.873 for SVC, and from 0.810 to 0.815 for GNB. F1-score also increased from 0.764 to 0.600 for LR, from 0.843 to 0.883 for KNN, from 0.819 to 0.862 for SVC, and from 0.790 to 0.815 for GNB.