sigmoidF1: A Smooth F1 Score Surrogate Loss for Multilabel Classification

Bénédict, Gabriel; Vincent, Koops,; Odijk, Daan; Rijke, Maarten de

doi:10.48550/arxiv.2108.10566

Cited by 5 publications

(7 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Gamma correction helped in making the distinguishable features more evident, which was also evident in similar other studies [23]. A better understanding of the machine-learning performance can be analyzed using the F1-score, which is calculated using precision and sensitivity, especially for multiclass problems [64]. The best performing combination was the ResNet50 network using the original thermogram images with a weighted F1-score of 76.66%, followed by the MobileNetv2 network (75.74%) using the AHE-enhanced thermogram, ResNet18 using the original thermogram (75.61%), and ResNet18 and ResNet50 using the Gamma-enhanced thermogram provided scores of 74.41%, and 74.17%, respectively.…”

Section: Discussionsupporting

confidence: 53%

Thermal Change Index-Based Diabetic Foot Thermogram Image Classification Using Machine Learning Techniques

Khandakar

Chowdhury

Reaz

et al. 2022

Sensors

View full text Add to dashboard Cite

Diabetes mellitus (DM) can lead to plantar ulcers, amputation and death. Plantar foot thermogram images acquired using an infrared camera have been shown to detect changes in temperature distribution associated with a higher risk of foot ulceration. Machine learning approaches applied to such infrared images may have utility in the early diagnosis of diabetic foot complications. In this work, a publicly available dataset was categorized into different classes, which were corroborated by domain experts, based on a temperature distribution parameter—the thermal change index (TCI). We then explored different machine-learning approaches for classifying thermograms of the TCI-labeled dataset. Classical machine learning algorithms with feature engineering and the convolutional neural network (CNN) with image enhancement techniques were extensively investigated to identify the best performing network for classifying thermograms. The multilayer perceptron (MLP) classifier along with the features extracted from thermogram images showed an accuracy of 90.1% in multi-class classification, which outperformed the literature-reported performance metrics on this dataset.

show abstract

Section: Discussionsupporting

confidence: 53%

Thermal Change Index-Based Diabetic Foot Thermogram Image Classification Using Machine Learning Techniques

Khandakar

Chowdhury

Reaz

et al. 2022

Sensors

View full text Add to dashboard Cite

show abstract

“…Considering LSTM, in the case of the wrapping machine the only hyperparameters that vary are the type of LSTM layer, Unidirectional or Bidirectional, and the loss function. For the latter, we compare the sigmoidF 1 loss with β = 1 and η = 0 (from now on referred to as F 1 loss) [77] and the Binary Cross-Entropy (BCE) loss. The use of BiLSTM has been shown to be effective in the works [74,78], which focus on forecasting problems.…”

Section: Algorithms and Hyperparameter Tuningmentioning

confidence: 99%

“…The use of BiLSTM has been shown to be effective in the works [74,78], which focus on forecasting problems. The two compared loss functions have been proven to deliver the top performances in [77]. Each loss function serves a distinct purpose and offers advantages and disadvantages.…”

Section: Algorithms and Hyperparameter Tuningmentioning

confidence: 99%

Predicting Machine Failures From Multivariate Time Series: An Industrial Case Study

Pinciroli Vago,

Forbicini,

Fraternali

2024

Preprint

View full text Add to dashboard Cite

Non-neural Machine Learning (ML) and Deep Learning (DL) are used to predict system failures in industrial maintenance. However, only a few studies have assessed the effect of varying the amount of past data used to make a prediction and the extension in the future of the forecast. This study evaluates the impact of the size of the reading window and of the prediction window on the performances of models trained to forecast failures in three data sets of (1) an industrial wrapping machine working in discrete sessions, (2) an industrial blood refrigerator working continuously and (3) a nitrogen generator working continuously. A binary classification task assigns the positive label to the prediction window based on the probability of a failure to occur in such an interval. Six algorithms (logistic regression, random forest, support vector machine, LSTM, ConvLSTM and Transformers) are compared using multivariate time series. The dimension of the prediction windows plays a crucial role and the results highlight the effectiveness of DL approaches in classifying data with diverse time-dependent patterns preceding a failure and the effectiveness of ML approaches in classifying similar and repetitive patterns preceding a failure.

show abstract

“…Surrogate loss functions attempt to mimic certain aspects of the F β and is another related area. For example, sigmoidF1 from [1] creates smooth versions for the entries of the confusion matrix, which is used to create a differentiable loss function that imitates the F 1 . This smooth differentiability is another application of a sigmoid approximation similar to [12].…”

Section: Related Workmentioning

confidence: 99%

“…They also motivate the rationale on using statistical distributions to understand the CE curve. The van Rijsbergen's effectiveness measure is given in (1).…”

Section: Introductionmentioning

confidence: 99%

Reformulating van Rijsbergen's $F_β$ metric for weighted binary cross-entropy

Ramdhani¹

2022

Preprint

View full text Add to dashboard Cite

The separation of performance metrics from gradient based loss functions may not always give optimal results and may miss vital aggregate information. This paper investigates incorporating a performance metric alongside differentiable loss functions to inform training outcomes. The goal is to guide model performance and interpretation by assuming statistical distributions on this performance metric for dynamic weighting. The focus is on van Rijsbergen's Fβ metric -a popular choice for gauging classification performance. Through distributional assumptions on the Fβ, an intermediary link can be established to the standard binary cross-entropy via dynamic penalty weights. First, the Fβ metric is reformulated to facilitate assuming statistical distributions with accompanying proofs for the cumulative density function. These probabilities are used within a knee curve algorithm to find an optimal β or βopt. This βopt is used as a weight or penalty in the proposed weighted binary cross-entropy. Experimentation on publicly available data with imbalanced classes mostly yields better and interpretable results as compared to the baseline. For example, for the IMDB text data with known labeling errors, a 14% boost is shown. This methodology can accelerate training and provide better interpretation.

show abstract

sigmoidF1: A Smooth F1 Score Surrogate Loss for Multilabel Classification

Abstract: Multiclass multilabel classification refers to the task of attributing multiple labels to examples via predictions. Current models formulate a reduction of that multilabel set-

Cited by 5 publications

References 38 publications

Thermal Change Index-Based Diabetic Foot Thermogram Image Classification Using Machine Learning Techniques

Thermal Change Index-Based Diabetic Foot Thermogram Image Classification Using Machine Learning Techniques

Predicting Machine Failures From Multivariate Time Series: An Industrial Case Study

Reformulating van Rijsbergen's $F_β$ metric for weighted binary cross-entropy

Contact Info

Product

Resources

About