Active learning for imbalanced data under cold start

Abstract

Labeled data is essential in modern systems that rely on Machine Learning (ML) for predictive modelling. Such systems may suffer from the cold-start problem: supervised models work well but, initially, there are no labels, which are costly or slow to obtain. This problem is even worse in imbalanced data scenarios. Online financial fraud detection is an example where labeling is: i) expensive, or ii) it suffers from long delays, if relying on victims filing complaints. The latter may not be viable if a model has to be in place immediately, so an option is to ask analysts to label events while minimizing the number of annotations to control costs. We propose an Active Learning (AL) annotation system for datasets with orders of magnitude of class imbalance, in a cold start streaming scenario. We present a computationally efficient Outlier-based Discriminative AL approach (ODAL) and design a novel 3-stage sequence of AL labeling policies where it is used as warm-up. Then, we perform empirical studies in four real world datasets, with various magnitudes of class imbalance. The results show that our method can more quickly reach a high performance model than standard AL policies. Its observed gains over random sampling can reach 80% and be competitive with policies with an unlimited annotation budget or additional historical data (with 1/10 to 1/50 of the labels).

Anomaly Detection in Cryptocurrency Transactions With Active Learning

“…The authors emphasized the importance of aggregating transactions for capturing correlations and found that IF and RF were the best-performing ML models with high detection rates. Other studies have also contributed to the literature by designing multi-stage AL labeling policies (Lorenz et al, 2020;Barata et al, 2021), which initially apply unsupervised AD algorithms to rank the most anomalous transactions for review before switching to supervised AL policies. Barata et al (2021) proposed an intermediate stage using ODAL as a warm-up learner, which efficiently alleviated the cold start scenario with high-class imbalance.…”

“…Other studies have also contributed to the literature by designing multi-stage AL labeling policies (Lorenz et al, 2020;Barata et al, 2021), which initially apply unsupervised AD algorithms to rank the most anomalous transactions for review before switching to supervised AL policies. Barata et al (2021) proposed an intermediate stage using ODAL as a warm-up learner, which efficiently alleviated the cold start scenario with high-class imbalance. Both studies found that switching to supervised learners improved the models' performance, with Lorenz et al (2020) achieving promising results by matching the performance of a supervised baseline using just 5% of the labels.…”

“…Training high-performance supervised classifiers for FD is crucial, but the lack of labeled data makes it challenging to train such ML models (Labanca et al, 2022). The process of labeling an entire dataset requires a vast sample of manually reviewed transactions, which is often unpractical due to the high costs involved and the limited investigative time and budget available for companies (Barata et al, 2021). Thus, an efficient mechanism is needed to help analysts focus their investigation on the most relevant transactions with a higher probability of being fraudulent and to efficiently uncover increasingly complex and sophisticated fraud patterns, given that fraudsters are constantly finding new ways to commit fraudulent activities.…”

Active Learning in the Detection of Anomalies in Cryptocurrency Transactions

“…Assuming a cold start scenario, it is intended with this work to assess the feasibility of using Anomaly Detection (AD) algorithms [1,5] and Active Learning (AL) techniques [6] to uncover fraudulent patterns in cryptocurrency transactions [7].…”