Class-wise Calibration: A Case Study on COVID-19 Hate Speech

Abstract

Proper calibration of deep-learning models is critical for many high-stakes problems. In this paper, we show that existing calibration metrics fail to pay attention to miscalibration on individual classes, hence overlooking minority classes and causing significant issues on imbalanced classification problems. Using a COVID-19 hate-speech dataset, we first discover that in imbalanced datasets, miscalibration error on an individual class varies greatly, and error on minority classes can be magnitude times worse than what is suggested by the overall calibration performance. To address this issue, we propose a new metric based on expected miscalibration error, named as Contraharmonic Expected Calibration Error (CECE), which punishes severe miscalibration on individual classes. We further devise a novel variant of temperature scaling for imbalanced data to improve class-wise miscalibration, which re-weights the loss function by the inverse class count to tune the scaling parameter to reduce worst-case minority calibration error. Our case study on a benchmarking COVID-19 hate speech task shows the effectiveness of our calibration metric and our temperature scaling strategy.

Region-dependent temperature scaling for certainty calibration and application to class-imbalanced token classification

“…Unlike vector and matrix scaling, RD-TS cannot change the relative ranking of logits, and therefore model accuracy is retained (in single-label settings). One line of future work could be to apply RD-TS on top of weighted temperature scaling, a method known to decrease variance in calibration error among classes (Obadinma et al, 2021). Another line of work would be to investigate whether improved certainty estimates can increase model accuracy (in multi-label settings where predictions are applied by meeting a certainty threshold), especially in out-of-domain problems.…”

“…Weighted temperature scaling (WTS): TS using a class-weighted NLL loss during convergence (Obadinma et al, 2021).…”

A survey on confidence calibration of deep learning under class imbalance data

“…To prevent the model trained on a long-tailed dataset from being skewed to high-frequency classes, Islam et al proposed a class distribution-aware temperature scaling, which uses class frequency information to get the temperature value. Similarly, Weighted temperature scaling is proposed by Qkadinma et al [149], which re-weights the loss function by the inverse class count to tune the scaling parameter.…”

“…T ransf er Knowledge f rom Head to T ail [16] ⃝ ✕ ⃝ △ ✕ ⃝ ⃝ Scipy 20 Class-wise Loss Scaling [150] △ ✕ △ △ △ ⃝ ⃝ PyTorch 21 Region-dependent T emperature Scaling [143] ⃝ ✕ ⃝ △ ✕ ⃝ ⃝ URL invalid 22 Class-wise T emperature Scaling [144] ⃝ ✕ △ △ △ ⃝ ⃝ N/A N ormalized Calibration [17] △ △ △ ⃝ ✕ ⃝ △ PyTorch 23 Gaussian and Gamma Calibration [151] ⃝ ✕ △ ⃝ ✕ ✕ ⃝ URL invalid 24 CS-T S-AT C [145] △ ✕ △ △ △ ⃝ ⃝ PyTorch 25 Dual-Branch T emperature Scaling [148] △ ✕ △ △ △ ⃝ ⃝ N/A Class-distribution-aware T S [21] △ ✕ △ ⃝ △ ⃝ ⃝ URL invalid 14 W eighted T emperature Scaling [149] △…”

“…, where γ represents the fraction of the minority class. Obadinma et al [149] found that calibration error on minority classes may be magnitude times worse than that on all classes, so they proposed Contraharmonic expected calibration error (CECE), which performs contraharmonic mean on the ECE of all classes, the calculation formula is as follows:…”