Measuring the Prediction Difficulty of Individual Cases in a Dataset using Machine Learning

Abstract

Different levels of prediction difficulty are one of the key factors that researchers encounter when applying machine learning to data. Although previous studies have introduced various metrics for assessing the prediction difficulty of individual cases, these metrics require specific dataset preconditions. Additionally, evaluating these preconditions in real-world datasets can be challenging due to their diversity and complexity. In this paper, we propose three novel metrics for measuring the prediction difficulty of individual cases using fully-connected feedforward neural networks. The new metrics are designed to be universal metrics capable of calculating prediction difficulty in any dataset. The first metric is based on the complexity of the neural network needed to make a correct prediction. The second metric employs a pair of neural networks: one makes a prediction for a given case, and the other predicts whether the prediction made by the first model is likely to be correct. The third metric assesses the variability of the neural network’s predictions. We investigated these metrics using a variety of datasets, visualized their values, and compared them to existing metrics from the literature. The results demonstrate the effectiveness of our metrics across diverse datasets. We expect our metrics will provide researchers with a new perspective on understanding their datasets and applying machine learning in various fields.

Prediction Performance Metrics Considering the Difficulty of Individual Cases

“…Three case di culty calculation metrics were used to measure the di culty of individual cases in both the simulated and real-world datasets. For more details on these metrics, see our previous work [12].…”

“…Our performance metrics reward models for correct predictions on di cult cases and penalize them for incorrect predictions on easy cases. To calculate case di culty, we used three different case di culty metrics developed using neural networks (NNs) in our previous research [12]. We then compared the models' performance with and without considering case di culty.…”