Stop ordering machine learning algorithms by their explainability! A user-centered investigation of performance and explainability

Why Providing Humans with Interpretable Algorithms May, Counterintuitively, Lead to Lower Decision-making Performance

“…First, algorithms on average differ in their capability to predict the likelihood of a future event to happen and last (Bonde Thylstrup et al, 2019;Henriksen & Bechmann, 2020). For example, ML models perform better than rule-based models in high data-frequency contexts with high variability and turbulent conditions, or in changing contextual conditions that require learning and adaptation (Herm et al, 2022). Second, algorithms differ in their interpretability.…”

A nascent design theory for explainable intelligent systems

“…Recent examples can be found in all kind of application fields, such as medicine (McKinney et al 2020), manufacturing (Nor et al 2022, or social media . For the following, we align with Herm, Heinrich, et al (2022a) and Mohseni et al (2021) by referring to these types of AI-based DSS or intelligent DSS as intelligent systems.…”

“…According to definition of Berente et al (2021, 4), AI is the "frontier of computational advancements that references human intelligence in addressing ever more complex decision-making problems", which is pushed further by intelligent systems to provide decisionmaking with human-like or even superhuman cognitive abilities (Herm, Heinrich, et al, 2022a;Janiesch et al 2021). To enable these decision-making abilities for decision support, intelligent systems use ML to allow for the autonomous generation of decision knowledge based on observations (Nilsson 2014;Poole et al 1998).…”

“…On the downside, while their architectural structure is becoming more complex, the user's ability to comprehend the inner decision logic decreases (Ribeiro et al 2016). In practice, this results in a complex tradeoff between the performance and the explainability of these models (Herm, Heinrich, et al, 2022a). That is, models with high predictive accuracy also tend to be more challenging to comprehend and vice versa (Herm et al 2021).…”

“…Unsurprisingly many kinds of systems are using AI today to achieve or surpass human intelligence for selected tasks (Berente et al 2021). AI-based decision support systems (DSS) are a particular type of such systems capable of supporting human decision-making in many situations (Herm, Heinrich, et al, 2022a;Mohseni et al 2021) such as evaluating heat-flux sensor data to track plastic welding processes and ensure the durability of the welding seam (see Section 5).…”