Creating Unbiased Public Benchmark Datasets with Data Leakage Prevention for Predictive Process Monitoring

Abstract

Advances in AI, and especially machine learning, are increasingly drawing research interest and efforts towards predictive process monitoring, the subfield of process mining (PM) that concerns predicting next events, process outcomes and remaining execution times. Unfortunately, researchers use a variety of datasets and ways to split them into training and test sets. The documentation of these preprocessing steps is not always complete. Consequently, research results are hard or even impossible to reproduce and to compare between papers. At times, the use of non-public domain knowledge further hampers the fair competition of ideas. Often the training and test sets are not completely separated, a data leakage problem particular to predictive process monitoring. Moreover, test sets usually suffer from bias in terms of both the mix of case durations and the number of running cases. These obstacles pose a challenge to the field's progress. The contribution of this paper is to identify and demonstrate the importance of these obstacles and to propose preprocessing steps to arrive at unbiased benchmark datasets in a principled way, thus creating representative test sets without data leakage with the aim of levelling the playing field, promoting open science and contributing to more rapid progress in predictive process monitoring.

Prediction of amputation risk of patients with diabetic foot using classification algorithms: A clinical study from a tertiary center

“…Additionally, test sets are often biased, and these challenges present obstacles to the advancement of the field. 50 The process of developing predictive machine-learning models in this study involved using the training dataset, which comprised 80% of the total dataset, and optimizing various hyperparameters and parameters using methods via grid search.…”

“…It has been stated that in the field of predictive process monitoring, the separation between training and test sets is often incomplete, leading to a problem of data leakage. Additionally, test sets are often biased, and these challenges present obstacles to the advancement of the field 50 . The process of developing predictive machine‐learning models in this study involved using the training dataset, which comprised 80% of the total dataset, and optimizing various hyperparameters and parameters using methods via grid search.…”

What Averages Do Not Tell -- Predicting Real Life Processes with Sequential Deep Learning

“…We take the time attribute on the scale of seconds granularity and transform it to relative time expressing the duration of events (by the difference of two consecutive event timestamps). This also prevents a particular data leakage from the test set as indicated by [29]. The time attribute value corresponding to any special symbols for the activity label (e.g.…”

“…The performance of suffix prediction was evaluated only on average performance measures for a few real-life event logs [18]. In addition, different pre-processing and evaluation strategies have been used which makes it difficult to compare the results of the different approaches [18,29]. Our work contributes a unified framework for case suffix prediction with seven sequential DL models, multimodal event attribute fusion, and multi-target prediction (Section 3).…”

Explainable Artificial Intelligence for Improved Modeling of Processes

“…Previous work in PPM faces a few challenges: unclear training/test set splits and pre-processing schemes, which leads to noticeably different findings and challenges w.r.t reproducibility [33]. Since datasets often contain duplicates which are not respected by the evaluation schemes, results are often confounded.…”