Early Retrieval Problem and Link Prediction Evaluation via the Area Under the Magnified ROC

Abstract

Link prediction is an unbalanced early retrieval problem, whose goal is to prioritize a small cohort of positive links on top of a list largely populated by unlabelled links. Differently from binary classification, here the evaluation focuses on how the predictor prioritizes the positive class because, in practice, a negative class does not exist. Previous studies explained that AUC-ROC is not apt for unbalanced class problems and is misleading for early retrieval problems, therefore standard AUC-ROC is not appropriate for evaluation of link prediction. However, some scholars argue that an AUC-ROC like evaluation accounting for the relative positioning of the few positive links among the vastness of unlabelled links remains a valid concept to pursue. Here we propose the area under the magnified ROC (AUC-mROC), a new measure that adjusts the standard AUC-ROC to work also for unbalanced early retrieval problems such as link prediction.

“Stealing fire or stacking knowledge” by machine intelligence to model link prediction in complex networks

“… 1 about near optimality are based on evaluations using AUC-ROC, although several recent studies have shown that AUC-ROC is inappropriate for early retrieval problems and for unbalanced prediction tasks, 9 , 14 , 15 , 16 , 17 , 18 , 19 , 20 such as link prediction. A recent study of Muscoloni and Cannistraci 21 proposes the AUC-mROC that we report in the main figures of this study (in lieu of AUC-ROC whose values are provided for completeness in Tables 1 and 2 of this study), which adjusts the evaluation issues largely commented in the literature about the AUC-ROC. In conclusion, the precision-based results in Table 1 of Ghasemian et al., 1 as well as the results in our study, are rising concrete concerns about the near optimality of stacking models when the link prediction task is evaluated according to more appropriate evaluation measures.…”

“…AUC-mROC can adjust the AUC-ROC for evaluation of early retrieval problems in general and link prediction in particular. 21 For this reason, we prefer to report AUC-mROC in the main figures of this study, and AUC-ROC in the Tables 1 and 2 .…”

“… 2 https://github.com/biomedical-cybernetics/stealing_fire_or_stacking_knowledge_to_model_link_prediction SBM Peixoto 7 https://graph-tool.skewed.de/ ; https://github.com/biomedical-cybernetics/stealing_fire_or_stacking_knowledge_to_model_link_prediction Stacking basic Ghasemian et al. 1 ; This paper https://github.com/Aghasemian/OptimalLinkPrediction ; https://github.com/biomedical-cybernetics/stealing_fire_or_stacking_knowledge_to_model_link_prediction Stacking basic + CH + SPM + SBM This paper https://github.com/biomedical-cybernetics/stealing_fire_or_stacking_knowledge_to_model_link_prediction Evaluation measures Muscoloni and Cannistraci 21 https://github.com/biomedical-cybernetics/stealing_fire_or_stacking_knowledge_to_model_link_prediction ; https://github.com/biomedical-cybernetics/AUC-mROC Other Real-networks dataset Ghasemian et al. 1 ; This paper https://github.com/Aghasemian/OptimalLinkPrediction ; https://github.com/biomedical-cybernetics/stealing_fire_or_stacking_knowledge_to_model_link_prediction …”

Comparing discriminating abilities of evaluation metrics in link prediction

“…Well-know threshold-free metrics include BP [31], AUC [34], AUPR [35], and NDCG [36]. This work will also analyze a recently proposed metric called AUC-mROC [28]. BP represents the intersection of the Precision@k and Recall@k curves, specifically when the threshold k equals the size of the testing set (i.e.…”

“…Additionally, Lobo et al [27] also questioned AUC and offered reasons against its use. In the pursuit of how to evaluate algorithms for imbalanced classification and a nuanced characterization of the differences between algorithms, some researchers introduced innovative evaluation metrics, such as area under the magnified receiver operating characteristic (AUC-mROC) [28].…”