No abstract

With a plethora of available classification performance measures, choosing the right metric for the right task requires careful thought. To make this decision in an informed manner, one should study and compare general properties of candidate measures. However, analysing measures with respect to complete ranges of their domain values is a difficult and challenging task. In this study, we attempt to support such analyses with a specialized visualization technique, which operates in a barycentric coordinate system using a 3D tetrahedron. Additionally, we adapt this technique to the context of imbalanced data and put forward a set of properties which should be taken into account when selecting a classification performance measure. As a result, we compare 22 popular measures and show important differences in their behaviour. Moreover, for parametric measures such as the F β and IBA α (G-mean), we analytically derive parameter thresholds that change measure properties. Finally, we provide an online visualization tool that can aid the analysis of complete domain ranges of performance measures. NOTICE: This is a preliminary version of an article submitted to Information Sciences all situations. However, which measure is used in a given problem seems to be, to a large extent, dictated simply by the measure's popularity rather than a thorough discussion of its properties.Although there are a few systematic studies on different properties of classifier performance measures [19,16,11,30], we still postulate the need for thorough analysis of the measures' behaviour. In particular, methods for: interpreting and comparing measures with respect to whole domain ranges, analysing their nature for different class and prediction distributions, and detecting the presence of unusual values are much needed. Theoretical investigations of these aspects are often very laborious and time consuming, especially when multi-dimensional aspects, provided by the confusion matrices, need to be taken into account. Due to these difficulties, such an analysis could be alternatively carried out with visual techniques to aid the understanding and interpretability of various measure properties.In this paper, we put forward a new visualization technique for analysing entire domains of classification performance measures, which depicts all possible configurations of predictions in a confusion matrix, regardless of the used classifier. For this purpose, we adapt an approach originally created for rule interestingness measures to the context of classification [31]. Contrary to existing performance measure visualizations, such as ROC space [11], the proposed approach presents measures in a space which is defined directly on elements of the confusion matrix, is easily interpretable in 3D, and remains defined for all elements of the domain. Moreover, based on the devised visualization, we propose ten properties which should be taken into account while selecting evaluation measures, particularly for class imbalanced data. Consequently, we compare 22 popular c...

Confirmation is a useful concept for assessing the impact of the premise on the conclusion of a rule induced from data. Interpretation of probabilistic relationships between premise and conclusion of a rule led to four mathematical formulations of confirmation, called perspectives. The logical equivalence of these perspectives and the resulting general definition of confirmation underline the known qualitative aspect of the concept of confirmation. The quantitative aspect of confirmation is handled by definitions of particular confirmation measures. In this paper, we relate the qualitative and quantitative aspects by introducing a property of monotonicity of measures with respect to left-and right-hand side probabilities defining the perspectives. This new property permits consideration of confirmation measures in association with particular perspectives. We also identify several other properties that valuable confirmation measures should possess. A particular care is devoted to discussion of behavior of confirmation measures monotonic in different perspectives with respect to symmetry properties, taking also into account two new perspectives of Bayesian confirmation. We also prove that confirmation measures monotonic in the six perspectives are exhaustive in the sense that their set is closed under transformations related to symmetry properties. Finally, we verify which confirmation measures enjoy these properties.

Abstract. The work is devoted to multicriteria approaches to rule evaluation. It analyses desirable properties (in particular the property M, property of confirmation and hypothesis symmetry) of popular interestingness measures of decision and association rules. Moreover, it analyses relationships between the considered interestingness measures and enclosure relationships between the sets of non-dominated rules in different evaluation spaces. It's main result is a proposition of a multicriteria evaluation space in which the set of non-dominated rules will contain all optimal rules with respect to any attractiveness measure with the property M. By determining the area of rules with desirable value of a confirmation measure in the proposed multicriteria evaluation space one can narrow down the set of induced rules only to the valuable ones. Furthermore, the work presents an extension of an apriori-like algorithm for generation of rules with respect to attractiveness measures possessing valuable properties and shows some applications of the results to analysis of rules induced from exemplary datasets.

The paper focuses on Bayesian confirmation measures used for evaluation of rules induced from data. To distinguish between many confirmation measures, their properties are analyzed. The article considers a group of symmetry properties. We demonstrate that the symmetry properties proposed in the literature focus on extreme cases corresponding to entailment or refutation of the rule's conclusion by its premise, forgetting intermediate cases. We conduct a thorough analysis of the symmetries regarding that the confirmation should express how much more probable the rule's hypothesis is when the premise is present rather than when the negation of the premise is present. As a result we point out which symmetries are desired for Bayesian confirmation measures. Next, we analyze a set of popular confirmation measures with respect to the symmetry properties and other valuable properties, being monotonicity M, Ex 1 and weak Ex 1 , logicality L and weak L. Our work points out two measures to be the most meaningful ones regarding the considered properties.