Density-based weighting for imbalanced regression

Abstract

In many real world settings, imbalanced data impedes model performance of learning algorithms, like neural networks, mostly for rare cases. This is especially problematic for tasks focusing on these rare occurrences. For example, when estimating precipitation, extreme rainfall events are scarce but important considering their potential consequences. While there are numerous well studied solutions for classification settings, most of them cannot be applied to regression easily. Of the few solutions for regression tasks, barely any have explored cost-sensitive learning which is known to have advantages compared to sampling-based methods in classification tasks. In this work, we propose a sample weighting approach for imbalanced regression datasets called DenseWeight and a cost-sensitive learning approach for neural network regression with imbalanced data called DenseLoss based on our weighting scheme. DenseWeight weights data points according to their target value rarities through kernel density estimation (KDE). DenseLoss adjusts each data point’s influence on the loss according to DenseWeight, giving rare data points more influence on model training compared to common data points. We show on multiple differently distributed datasets that DenseLoss significantly improves model performance for rare data points through its density-based weighting scheme. Additionally, we compare DenseLoss to the state-of-the-art method SMOGN, finding that our method mostly yields better performance. Our approach provides more control over model training as it enables us to actively decide on the trade-off between focusing on common or rare cases through a single hyperparameter, allowing the training of better models for rare data points.

Machine Learning to Predict the Antimicrobial Activity of Cold Atmospheric Plasma-Activated Liquids

“…The dataset was resampled by using SMOGN as a data preprocessing step before the training of the regression models because the data distribution was not balanced. The SMOGN algorithm, which mixes oversampling with Gaussian noise [57], is based on the SMOTER method. The density of the outcome of the dataset after oversampling with the SMOGN approach is shown in Figure 7 together with the initial data density.…”

“…The SMOTER and the introduction of Gaussian Noise are two oversampling techniques combined in the SMOGN approach. Based on the distance to the k-nearest neighbors, SMOGN iterates through all unusual samples and selects between SMOTER's interpolation-based oversampling and Gaussian noise-based oversampling [57]. The data imbalance effect was eliminated by using the SMOTE and SMOGN techniques for classification and regression models, respectively to resample the dataset into the classes and change the relative frequency of the other labels.…”

Comparing Multiple Linear Regression, Deep Learning and Multiple Perceptron for Functional Points Estimation

“…The data augmentation and class weight approaches are standards in deep learning. Data augmentation is an approach to increasing the diversity of training data [55], and class weight is used to determine the weight of each categorical variable when the dataset is unbalanced [56]. These approaches might be employed to ensure the diversity and completeness of selected projects in the imbalanced projects.…”

Taming the Long Tail of Deep Probabilistic Forecasting

“…Many approaches revolve around modifications of SMOTE such as adapted to regression SMOTER [38], augmented with Gaussian Noise SMOGN [39], or [40] work extending for prediction of extremely rare values. [41] proposed DenseWeight, a method based on Kernel Density Estimation for better assessment of the relevance function for sample reweighing. [42] proposed a distribution smoothing over label (LDS) and feature space (FDS) for imbalanced regression.…”