Evaluating Conformance Measures in Process Mining Using Conformance Propositions

Abstract

Process mining sheds new light on the relationship between process models and real-life processes. Process discovery can be used to learn process models from event logs. Conformance checking is concerned with quantifying the quality of a business process model in relation to event data that was logged during the execution of the business process. There exist different categories of conformance measures. Recall, also called fitness, is concerned with quantifying how much of the behavior that was observed in the event log fits the process model. Precision is concerned with quantifying how much behavior a process model allows for that was never observed in the event log. Generalization is concerned with quantifying how well a process model generalizes to behavior that is possible in the business process but was never observed in the event log. Many recall, precision, and generalization measures have been developed throughout the years, but they are often defined in an ad-hoc manner without formally defining the desired properties up front. To address these problems, we formulate 21 conformance propositions and we use these propositions to evaluate current and existing conformance measures. The goal is to trigger a discussion by clearly formulating the challenges and requirements (rather than proposing new measures). Additionally, this paper serves as an overview of the conformance checking measures that are available in the process mining area. the years, many process discovery algorithms have been developed, producing process models in various forms, such as Petri nets, process trees, and BPMN.Event logs are often incomplete, i.e., they only contain a sample of all possible behavior in the business process. This not only makes process discovery challenging; it is also difficult to assess the quality of the process model in relation to the log. Process discovery algorithms take an event log as input and aim to output a process model that satisfies certain properties, which are often referred to as the four quality dimensions [1] of process mining: (1) recall: the discovered model should allow for the behavior seen in the event log (avoiding "non-fitting" behavior), (2) precision: the discovered model should not allow for behavior completely unrelated to what was seen in the event log (avoiding "underfitting"), (3) generalization: the discovered model should generalize the example behavior seen in the event log (avoiding "overfitting"), and (4) simplicity: the discovered model should not be unnecessarily complex. The simplicity dimension refers to Occam's Razor: "one should not increase, beyond what is necessary, the number of entities required to explain anything". In the context of process mining, this is often operationalized by quantifying the complexity of the model (number of nodes, number of arcs, understandability, etc.). We do not consider the simplicity dimension in this paper, since we focus on behavior and abstract from the actual model representation. Recall is often referred to as fitness in process min...

Adversarial System Variant Approximation to Quantify Process Model Generalization

“…The importance of generalization, i.e. measuring how well a process model represents the underlying system, is wellknown [6], [8], [14]. This stems from the difficulty of deriving the unknown system from a observation sample of limited size.…”

“…Therefore, process models that add new behavior without introducing new states are preferred and considered well-generalizing. However, this method ignores whether a behavior has not been observed in an event log [8].…”

“…Dongen et al [7] describe generalization as "how well the model explains unobserved system behavior". Similarly, Syring et al [8] describe generalization as a metric that is "concerned with quantifying how well a process model generalizes to behavior that is possible in the (...) process but was never observed". It is obvious that assessing the generalization metric is difficult since unobserved behavior is naturally unknown.…”

“…Process models are used to draw conclusions about the underlying system and future behavior [9]. Generalization, as an assessment metric, cannot be ignored in evaluating process models, because a process model that is solely evaluated on fitness, precision, and simplicity metrics might represent the observed recordings only [8], and hence overfits. Such a process model is inappropriate to derive conclusions about the system behavior.…”

“…Such a process model is inappropriate to derive conclusions about the system behavior. Recent research studies [8], [10] have shown that state-of-the-art methods fail in assessing the extent to which a process model generalizes unobserved behavior. Hence, the measurement of generalization remains unsolved.…”