A Robust profit measure for binary classification model evaluation

Abstract

Using profit-based evaluation measures is a necessity in businessoriented contexts, as they aid companies in making cost-optimal decisions. Among the measures that effectively include the true nature of costs and benefits in binary classification, the expected maximum profit (EMP) has been used successfully for churn prediction and credit scoring, and defined in general for binary classification problems. However, despite its competitive results against the most frequently used measures, the EMP relies on a fixed probability distribution of costs and benefits, the range of which in real applications is not entirely known. In this paper, we propose to extend this measure by adding random shocks to these distributions. We call this new measure the R-EMP, following the convention of the analogous EMP measure. Our metric adds a stochastic component to each point of * NOTICE: this is the author's version of a work that was accepted for publication in Expert Systems with Applications in September 19, 2017, published online as a self-archive copy after the 24 month embargo period. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. Please cite this paper as follows: Franco Garrido, Wouter Verbeke, Cristin Bravo, A Robust Profit Measure for Binary Classification Model Evaluation, In Expert Systems with Applications, 2017, Accepted: Available Online https://doi.org/10.1016/j.eswa.2017.09.045. 1 the cost-benefit distributions, assuming that costs and benefits have a fixed probability, but its distribution range is subject to an external shock, which can be different for each cost or benefit. The experimental set-up is focused on a credit scoring application using a dataset of a Chilean financial institution, with the attribute selection for a logistic regression being accomplished using the AUC, EMP, H-measure, and R-EMP as the selection criteria. The results indicate that the R-EMP measure is the most robust metric for achieving the greatest profit for the company under uncertain external conditions.

Comparison of Supervised Classification Techniques With Alos Palsar Sensor for Roorkee Region of Uttarakhand, India

“…In unsupervised algorithms, the user is not aware of the initial conditions of the data, field or the area of observation. The supervised classification techniques are categorized as parallelepiped [3] minimum distance [4], mahalanobis distance [5], , maximum likelihood [6], spectral angle mapper [7], spectral information divergence, [8], binary encoding [9], artificial neural network, [10] and support vector machine [11]. Unsupervised classification techniques include Isodata, [12] and kmeans algorithm [13].…”

Cervical Cancer Prediction through Different Screening Methods using Data Mining

“…Boosted decision tree, decision forest and decision jungle algorithms were used to determine the prediction ability of tested models by computing the accuracy, sensitivity, specificity and AUROC curve. AUROC curve is a best measure to evaluate the performance of classification models [39][40][41][42]. The AUROC curve performance of proposed models has shown in Fig.…”

“…AUROC curve was also used for evaluation of different techniques [18,27] in biomedical data mining. There are 50% of cervical cancer identification in females age (35)(36)(37)(38)(39)(40)(41)(42)(43)(44)(45)(46)(47)(48)(49)(50)(51)(52)(53)(54) and around 20% diagnosed more than 65 years old as well as around 15% of between the age of (20 -30). Median age for diagnosis in cervical cancer is 48 years.…”

Clues from networks: quantifying relational risk for credit risk evaluation of SMEs

“…Under a given granting ratio, lower default rate means less losses and more benefit for financial institutions. Notably, the economic benefit of additional features in credit risk evaluation could also be evaluated by other measures (Maldonado et al 2017;Garrido et al 2018).…”