Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs

Abstract

As the use of machine learning (ML) models in product development and data-driven decision-making processes became pervasive in many domains, people's focus on building a well-performing model has increasingly shifted to understanding how their model works. While scholarly interest in model interpretability has grown rapidly in research communities like HCI, ML, and beyond, little is known about how practitioners perceive and aim to provide interpretability in the context of their existing workflows. This lack of understanding of interpretability as practiced may prevent interpretability research from addressing important needs, or lead to unrealistic solutions. To bridge this gap, we conducted 22 semi-structured interviews with industry practitioners to understand how they conceive of and design for interpretability while they plan, build, and use their models. Based on a qualitative analysis of our results, we differentiate interpretability roles, processes, goals and strategies as they exist within organizations making heavy use of ML models. The characterization of interpretability work that emerges from our analysis suggests that model interpretability frequently involves cooperation and mental model comparison between people in different roles, often aimed at building trust not only between people and models but also between people within the organization. We present implications for design that discuss gaps between the interpretability challenges that practitioners face in their practice and approaches proposed in the literature, highlighting possible research directions that can better address real-world needs.

Bridging the Gap Between Ethics and Practice

“…They argue that explainability helps designers enhance correctness, identify improvements in training data, account for changing realities, support users in taking control, and increase user acceptance. An interview study with 22 machine learning professionals documented the value of explainability for developers, testers, managers, and users [52]. However, explainability methods are only slowly finding their way into widely used applications and possibly in ways that are different from the research.…”

How Teams Communicate about the Quality of ML Models: A Case Study at an International Technology Company

“…As machine learning is becoming an integral part of our lives, researchers have been investigating the biases that could arise and how to mitigate them [5,44]. Work on bias mitigation looked into the interpretability and transparency of these models [27,59,60], what industry practitioners need to improve the fairness in ML systems [29,30], and the perceived fairness of biased algorithms in current practices [26,51]. In our work, we looked into how much bias and fairness in algorithms is communicated as an aspect of ML models' quality and who within teams and organizations is interested in this.…”

Adoption-Driven Data Science for Transportation Planning: Methodology, Case Study, and Lessons Learned

“…Two examples include "the degree to which an observer can understand the cause of a decision" [22] and "a method is interpretable if a user can correctly and efficiently predict the method's result" [23]. Furthermore, different roles also have different interpretations [24]. Rather than delimiting what interpretability is, we propose to reach an agreement on its meaning for all project stakeholders.…”