Pitfalls of Assessing Extracted Hierarchies for Multi-Class Classification

Abstract

Using hierarchies of classes is one of the standard methods to solve multi-class classification problems. In the literature, selecting the right hierarchy is considered to play a key role in improving classification performance. Although different methods have been proposed, there is still a lack of understanding of what makes one method to extract hierarchies perform better or worse.To this effect, we analyze and compare some of the most popular approaches to extracting hierarchies. We identify some common pitfalls that may lead practitioners to make misleading conclusions about their methods. In addition, to address some of these problems, we demonstrate that using random hierarchies is an appropriate benchmark to assess how the hierarchy's quality affects the classification performance.In particular, we show how the hierarchy's quality can become irrelevant depending on the experimental setup: when using powerful enough classifiers, the final performance is not affected by the quality of the hierarchy. We also show how comparing the effect of the hierarchies against non-hierarchical approaches might incorrectly indicate their superiority.Our results confirm that datasets with a high number of classes generally present complex structures in how these classes relate to each other. In these datasets, the right hierarchy can dramatically improve classification performance.

Constructing Multiclass Classifiers using Binary Classifiers Under Log-Loss

“…In this case the logistic regression loss of (18) would be: Remark 2. Unlike in the 0-1 loss case [16], all trees that induce the same probability P , have identical loglosses. When using mismatched binary classifiers on the tree nodes, different structures might yield preferable regrets.…”

“…For the 0 − 1 loss, many works have developed methods for constructing multiclass classifiers from binary ones [7]- [16] and have studied the dependence of the overall error probability on the error probabilities of the binary classifiers. Nevertheless, to the best of our knowledge, this is the first work to address this topic under log-loss.…”

“…Another basic method for constructing a multiclass classifier from binary elements is by using a hierarchical tree structure [12]- [16]. In such a tree, every node represents a partition of a set of input classes into two disjoint subsets (not necessarily of an equal size).…”

Hierarchical Multi-Fault Prognostics for Complex Systems

“…Our first step is to evaluate our approach on a randomly generated hierarchy. Comparing against a random hierarchy gives us an adequate benchmark to validate the method to extract hierarchy [9]. At each fold, we will draw a new hierarchy, train, evaluate the models for each node, and select the combination of models that provide useful results, similar to how we did in the previous subsection.…”