Optimality Analysis of Boundary-Uncertainty-Based Classifier Model Parameter Status Selection Method

Ha, David; Watanabe, H.; Tomotoshi, Yuya; Delattre, Émilie; Katagiri, Shigeru

doi:10.1145/3297067.3297076

Cited by 2 publications

(3 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Substituting (17), (18) and using the Bayes theorem in Eq. 16 gives (19) where we note that B * (k, l) is defined in Eq. 14 and λ U C i(x;Λ)j (x;Λ) (x; Λ) |C kl is −U(x; Λ) in most cases.…”

Section: Expected Boundary Uncertaintymentioning

confidence: 99%

“…The limitations described above come from the intrinsic difficulty of estimating the error probability. To circumvent these limitations, we proposed finding the optimally trained classifier through a new classifier evaluation metric that is uniquely easy to estimate in principle, which we termed "boundary uncertainty" or alternately "Bayes-boundaryness" [16][17][18]. We chose the name "boundary uncertainty" because this evaluation metric measures the generalization ability of a classifier based on how equal class posterior probabilities are along the classifier boundary.…”

Section: Introductionmentioning

confidence: 99%

“…In addition to defining boundary uncertainty, our previous work also proposed a boundary uncertainty estimation method that here we refer to as "Proposal 1." Through experiments and theoretical analysis, we showed that Proposal 1 could perform classifier evaluation using neither sample distribution assumptions nor data resampling methods [16][17][18]. However, Proposal 1 relied on largely unclear settings and heavy treatment that seriously limited both its accuracy and its scalability.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An Improved Boundary Uncertainty-Based Estimation for Classifier Evaluation

Katagiri

Watanabe

et al. 2021

J Sign Process Syst

Self Cite

View full text Add to dashboard Cite

This paper proposes a new boundary uncertainty-based estimation method that has significantly higher accuracy, scalability, and applicability than our previously proposed boundary uncertainty estimation method. In our previous work, we introduced a new classifier evaluation metric that we termed “boundary uncertainty.” The name “boundary uncertainty” comes from evaluating the classifier based solely on measuring the equality between class posterior probabilities along the classifier boundary; satisfaction of such equality can be described as “uncertainty” along the classifier boundary. We also introduced a method to estimate this new evaluation metric. By focusing solely on the classifier boundary to evaluate its uncertainty, boundary uncertainty defines an easier estimation target that can be accurately estimated based directly on a finite training set without using a validation set. Regardless of the dataset, boundary uncertainty is defined between 0 and 1, where 1 indicates whether probability estimation for the Bayes error is achieved. We call our previous boundary uncertainty estimation method “Proposal 1” in order to contrast it with the new method introduced in this paper, which we call “Proposal 2.” Using Proposal 1, we performed successful classifier evaluation on real-world data and supported it with theoretical analysis. However, Proposal 1 suffered from accuracy, scalability, and applicability limitations owing to the difficulty of finding the location of a classifier boundary in a multidimensional sample space. The novelty of Proposal 2 is that it locally reformalizes boundary uncertainty in a single dimension that focuses on the classifier boundary. This convenient reduction with a focus toward the classifier boundary provides the new method’s significant improvements. In classifier evaluation experiments on Support Vector Machines (SVM) and MultiLayer Perceptron (MLP), we demonstrate that Proposal 2 offers a competitive classifier evaluation accuracy compared to a benchmark Cross Validation (CV) method as well as much higher scalability than both CV and Proposal 1.

show abstract

Section: Expected Boundary Uncertaintymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%