A direct measure of discriminant and characteristic capability for classifier building and assessment

Abstract

Performance measures are used in various stages of the process aimed at solving a classifi-\ud cation problem. Unfortunately, most of these measures are in fact biased, meaning that they\ud strictly depend on the class ratio – i.e. on the imbalance between negative and positive samples.\ud After pointing to the source of bias for the best known measures, novel unbiased measures\ud are defined which are able to capture the concepts of discriminant and characteristic capability.\ud The combined use of these measures can give important information to researchers\ud involved in machine learning or pattern recognition tasks, in particular for classifier performance\ud assessment and feature selection

Stopwords Identification by Means of Characteristic and Discriminant Analysis

“…It has been proved that the ϕ − δ space is constrained by a rhomboidal shape (Armano, 2014), as reported in Figure 1…”

“…In this paper, we adopt two novel metrics able to provide relevant information to researchers in several IR and ML tasks (Armano, 2014). The proposal consists in two unbiased metrics, i.e., independent from the imbalance between positive (P) and negative (N) samples.…”

“…The above metrics show the following behavior: δ is high when a classifier partitions a given set of samples in accordance with the corresponding class labels; on the other hand, ϕ is high when a classifier tends to cluster negative and positive samples together. Let us note that here the conceptualization of "characteristic property" affects all samples, regardless from the corresponding class label, whereas the classical definition adopted in ML focuses only on samples that belong to the main class (Armano, 2014).…”

Phi-Delta-Diagrams: Software Implementation of a Visual Tool for Assessing Classifier and Feature Performance

“…Let us start with a short introduction to ϕ, δ measures, which are the core functions for creating the corresponding diagrams (for deeper insights, please refer to References [17,21]). The standard definition of ϕ and δ is…”

“…In the work by Armano [17], a new kind of diagram is proposed, i.e., ϕ, δ diagrams, whose primary intended use is the assessment of classifier performance in terms of accuracy, bias, and variance. In fact, their deployment occurs in a 2D space, whose coordinates represent the accuracy (stretched in [−1, 1]) and the bias (on the y-axis and x-axis, respectively).…”