Comparative Analysis Between Macro and Micro-Accuracy in Imbalance Dataset for Movie Review Classification

Suhaimi, Nur Suhailayani; Othman, Zalinda; Yaakub, Mohd Ridzwan

doi:10.1007/978-981-19-2394-4_8

Cited by 3 publications

(4 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As indicated in Table 4 , our Bi-LSTM model’s hyperparameters were selected based on the highest results after we tested and compared the outputs of many classifiers using different learning paradigms (different optimal hyperparameters and architectures) [ 86 ]. Since the imbalanced dataset leads to overfitting in the performance of the model [ 87 , 88 , 89 ], consequently, the proposed detection models faced overfitting in their performance because the dataset used in this research is imbalanced (we did not take action to balance the dataset for the reasons referred to in the “Dataset Visualization” section). To prevent the performance of the proposed model from overfitting as well as to increase the generalization of the model to the new data [ 90 , 91 ], and to generate high detection accuracy [ 92 ], we used several techniques and methods, including reducing the complexity of model design [ 93 , 94 ], using weight regularization with L1 and L2 regularizers, where L1 is the sum of the absolute weights, and L2 is the sum of the squared weights [ 90 , 93 ], and adding a dropout layer [ 91 , 92 ].…”

Section: Methodsmentioning

confidence: 99%

Fake News Detection Model on Social Media by Leveraging Sentiment Analysis of News Content and Emotion Analysis of Users’ Comments

Hamed

Aziz

Yaakub

2023

Sensors

Self Cite

View full text Add to dashboard Cite

Nowadays, social media has become the main source of news around the world. The spread of fake news on social networks has become a serious global issue, damaging many aspects, such as political, economic, and social aspects, and negatively affecting the lives of citizens. Fake news often carries negative sentiments, and the public’s response to it carries the emotions of surprise, fear, and disgust. In this article, we extracted features based on sentiment analysis of news articles and emotion analysis of users’ comments regarding this news. These features were fed, along with the content feature of the news, to the proposed bidirectional long short-term memory model to detect fake news. We used the standard Fakeddit dataset that contains news titles and comments posted regarding them to train and test the proposed model. The suggested model, using extracted features, provided a high detection accuracy of 96.77% of the Area under the ROC Curve measure, which is higher than what other state-of-the-art studies offer. The results prove that the features extracted based on sentiment analysis of news, which represents the publisher’s stance, and emotion analysis of comments, which represent the crowd’s stance, contribute to raising the efficiency of the detection model.

show abstract

Section: Methodsmentioning

confidence: 99%

Fake News Detection Model on Social Media by Leveraging Sentiment Analysis of News Content and Emotion Analysis of Users’ Comments

Hamed

Aziz

Yaakub

2023

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…The author in [37] has carried out research on comparative analysis of macro and micro accuracy through a three-classifier approach, namely: Naïve Bayes (NB), Support Vector Machine (SVM), and Random Forest (RF) in the Movie Reviews dataset. In align previous research, this study proposes two additional classifiers, namely: k-Nearest Neighbors (KNN) and Long Short-Term Memory (LSTM), which will be tested on the Plant-Disease Relation (PDR) dataset.…”

Section: Introductionmentioning

confidence: 99%

“…In comparative analysis, it is necessary to evaluate the model using various performance metrics [5], and it is an interdisciplinary method that encompasses broad cross-sections of disciplines [1]. Comparative analysis is one way to solve the problem of model performance in classifying unbalanced datasets.…”

Section: Introductionmentioning

confidence: 99%

“…Each algorithm used will produce a different confusion matrix, this is caused by differences in methods for handling unbalanced data and classifier algorithms used. Several previous studies analyzed the balance of macro-F1 and micro-F1 [5], [8] only. In the case of imbalanced data, micro accuracy F1 may be higher than macro accuracy because the model may more easily predict the more dominant class with a higher level of accuracy [10].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Comparative Analysis using Various Performance Metrics in Imbalanced Data for Multi-class Text Classification

Riyanto¹,

Sitanggang²,

Djatna³

et al. 2023

IJACSA

View full text Add to dashboard Cite

Precision, Recall, and F1-score are metrics that are often used to evaluate model performance. Precision and Recall are very important to consider when the data is balanced, but in the case of unbalanced data the F1-score is the most important metric. To find out the importance of these metrics, a comparative analysis is needed in order to determine which metric is appropriate for the data being analyzed. This study aims to perform a comparative analysis of various evaluation metrics on unbalanced data in multi-class text classification. This study uses an unbalanced multi-class text dataset including: association, negative, cause of disease, and treatment of disease. This study involves five classifiers as algorithm-level approach, namely: Multinomial Naive Bayes, K-Nearest Neigbors, Support Vector Machine, Random Forest, and Long Short-Term Memory. Meanwhile, data-level approach, this study involves under sampling, over sampling, and synthetic minority oversampling technique. Several evaluation metrics used to evaluate model performance include Precision, Recall, and F1-score. The results show that the most suitable evaluation metric for use on unbalanced data depends on the purpose of use and the desired priority, including the classifier that is suitable for handling multi-class assignments on unbalanced data. The results of this study can assist practitioners in selecting evaluation metrics that are in accordance with the goals and application needs of multi-class text classification.

show abstract

Robust weighted general performance score for various classification scenarios

Pandey,

Bagri,

Gupta

et al. 2024

IDT

View full text Add to dashboard Cite

Traditionally, performance measures such as accuracy, recall, precision, specificity, and negative predicted value (NPV) have been used to evaluate a classification model’s performance. However, these measures often fall short of capturing different classification scenarios, such as binary or multi-class, balanced or imbalanced, and noisy or noiseless data. Therefore, there is a need for a robust evaluation metric that can assist business decision-makers in selecting the most suitable model for a given scenario. Recently, a general performance score (GPS) comprising different combinations of traditional performance measures (TPMs) was proposed. However, it indiscriminately assigns equal importance to each measure, often leading to inconsistencies. To overcome the shortcomings of GPS, we introduce an enhanced metric called the Weighted General Performance Score (W-GPS) that considers each measure’s coefficient of variation (CV) and subsequently assigns weights to that measure based on its CV value. Considering consistency as a criterion, we found that W-GPS outperformed GPS in the above-mentioned classification scenarios. Further, considering W-GPS with different weighted combinations of TPMs, it was observed that no demarcation of these combinations that work best in a given scenario exists. Thus, W-GPS offers flexibility to the user to choose the most suitable combination for a given scenario.

show abstract

Comparative Analysis Between Macro and Micro-Accuracy in Imbalance Dataset for Movie Review Classification

Cited by 3 publications

References 15 publications

Fake News Detection Model on Social Media by Leveraging Sentiment Analysis of News Content and Emotion Analysis of Users’ Comments

Fake News Detection Model on Social Media by Leveraging Sentiment Analysis of News Content and Emotion Analysis of Users’ Comments

Comparative Analysis using Various Performance Metrics in Imbalanced Data for Multi-class Text Classification

Robust weighted general performance score for various classification scenarios

Contact Info

Product

Resources

About