Mitigating Class-Boundary Label Uncertainty to Reduce Both Model Bias and Variance

Abstract

The study of model bias and variance with respect to decision boundaries is critically important in supervised learning and artificial intelligence. There is generally a tradeoff between the two, as fine-tuning of the decision boundary of a classification model to accommodate more boundary training samples (i.e., higher model complexity) may improve training accuracy (i.e., lower bias) but hurt generalization against unseen data (i.e., higher variance). By focusing on just classification boundary fine-tuning and model complexity, it is difficult to reduce both bias and variance. To overcome this dilemma, we take a different perspective and investigate a new approach to handle inaccuracy and uncertainty in the training data labels, which are inevitable in many applications where labels are conceptual entities and labeling is performed by human annotators. The process of classification can be undermined by uncertainty in the labels of the training data; extending a boundary to accommodate an inaccurately labeled point will increase both bias and variance. Our novel method can reduce both bias and variance by estimating the pointwise label uncertainty of the training set and accordingly adjusting the training sample weights such that those samples with high uncertainty are weighted down and those with low uncertainty are weighted up. In this way, uncertain samples have a smaller contribution to the objective function of the model’s learning algorithm and exert less pull on the decision boundary. In a real-world physical activity recognition case study, the data present many labeling challenges, and we show that this new approach improves model performance and reduces model variance.

Segmentation of PMSE Data Using Random Forests

“…This can imply that by employing weighted-down labels (discussed in Section 3.3), we can achieve a reduction in both model bias and variance, hence, leading to an improved performance. Although the performance gain achieved using weighted-down labels technique from the study by [6] is marginal, for further studies involving data labels beyond the three categories used in this paper, the gains achieved using the weighted-down labels technique could be significant.…”

“…This could be due to its poor generalization to new unseen data. Based on the segmentation results obtained in Figures 10-13, we can note that using the random forests for all filter sizes with mtry = 5 and weighted-down labels technique from the study by Almeida et al [6], it is possible to segment the data into the three different categories of interest. Furthermore, mtry = 5 is in line with the study by Probst et al [18], which suggests that the recommended number of predictors is approximately equal to the square root value of the total number of features, which is 35 in our case.…”

“…In their paper [6], Almeida et al propose a novel method that can reduce both model bias and model variance. In their method, first, estimation of the pixel-wise label uncertainty of training data is performed.…”

“…where C is the number of classes, k is the number of neighbors we consider for each sample (we employed k = 11 for our experiments), k y i is the number of neighbors with same label as x i , k j is the number of neighbors with class label y j , and d x i represents a vector with normalized distances to the k y i neighbors with the same class label as x i . For more details on the significance of the terms in the numerator and denominator of Equation (3), we refer the reader to the original paper [6]. Next, the training sample weights are adjusted such that the samples with high uncertainty are weighted-down and those with low uncertainty are weighted-up.…”

“…For our analysis, we employ random forests as a segmentation method. Next, we use a technique already developed by others [6] to select labeled data points. In this paper, we focus on segmenting PMSE signal in images of VHF (Very High Frequency) radar data from EISCAT (where EISCAT is the name of the research organization that operates the radar).…”