Evaluation of Classification Algorithms Using McDm and Rank Correlation

Abstract

Classification algorithm selection is an important issue in many disciplines. Since it normally involves more than one criterion, the task of algorithm selection can be modeled as multiple criteria decision making (MCDM) problems. Different MCDM methods evaluate classifiers from different aspects and thus they may produce divergent rankings of classifiers. The goal of this paper is to propose an approach to resolve disagreements among MCDM methods based on Spearman's rank correlation coefficient. Five MCDM methods are examined using 17 classification algorithms and 10 performance criteria over 11 public-domain binary classification datasets in the experimental study. The rankings of classifiers are quite different at first. After applying the proposed approach, the differences among MCDM rankings are largely reduced. The experimental results prove that the proposed approach can resolve conflicting MCDM rankings and reach an agreement among different MCDM methods.

Picture Hesitant Fuzzy Set and Its Application to Multiple Criteria Decision-Making

“…Table 15 shows that the Spearman's rank correlation coefficients between the proposed method and AHP and TOPSIS are 0.9722 and 0.9549, respectively, which demonstrate that the proposed method is highly correlated with these two methods. AHP and TOPSIS methods have been approved to be the most suitable two methods to solve web service selection problems [39]; thus, the comparison results above illustrate the feasibility of the proposed method. From the information aggregation of the proposed method, we can find that the calculating procedure of the proposed method is more complicated than AHP and TOPSIS methods.…”

“…Subsequently, the ranking results of different MCDM methods are presented in Table 14. The Spearman's rank correlation coefficient is a powerful tool for measuring the similarity between two MCDM methods [39]. Then, we can calculate the Spearman's rank correlation coefficients between the proposed method and the other five MCDM methods as shown in Table 15.…”

The BigGrams: the semi-supervised information extraction system from HTML: an improvement in the wrapper induction

“…In particular, in the machine learning field, we may enumerate several kinds of solutions to resolve the problems mentioned above. We may justify a statistic-based approach [57], and more general approach to build ranking method (a ranking-based approach) [2,40,41,79].…”

QoS based Web Service Selection and Multi-Criteria Decision Making Methods

“…A larger absolute value indicates a good agreement between one MCDM method and other MCDM method [13].…”