How can Explainability Methods be Used to Support Bug Identification in Computer Vision Models?

Abstract

Deep learning models for image classifcation sufer from dangerous issues often discovered after deployment. The process of identifying bugs that cause these issues remains limited and understudied. Especially, explainability methods are often presented as obvious tools for bug identifcation. Yet, the current practice lacks an understanding of what kind of explanations can best support the diferent steps of the bug identifcation process, and how practitioners could interact with those explanations. Through a formative study and an iterative co-creation process, we build an interactive design probe providing various potentially relevant explainability functionalities, integrated into interfaces that allow for fexible workfows. Using the probe, we perform 18 user-studies with a diverse set of machine learning practitioners. Two-thirds of the practitioners engage in successful bug identifcation. They use multiple types of explanations, e.g. visual and textual ones, through non-standardized sequences of interactions including queries and exploration. Our results highlight the need for interactive, guiding, interfaces with diverse explanations, shedding light on future research directions. CCS CONCEPTS• Human-centered computing → User interface programming; Empirical studies in HCI ; • Computing methodologies → Computer vision; • Software and its engineering → Software testing and debugging.

Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations

“…A detailed summary of types of undesired behaviors is listed in Table 6. In the undesired behavior detection, the effectiveness of explanations is evaluated by objective performance measures, such as the number of bugs identified [71], the share of participants that identify a certain bias [57,First Experiment] or by the deviations between model predictions and human predictions for unusual samples [53]. The perception of users regarding fair treatment by a system has primarily been researched in high-stakes applications such as granting loans [27] or granting bail for criminal offenders [73,74,75].…”

The Role of Human Knowledge in Explainable AI

“…Even though providing a wide variety of interactive explanations may contribute to improving the debugging of ML systems, it is still unclear which ones are the most useful. Seeking to answer such a question, Balayn et al [118] developed an interactive design probe that provides various explainability functionalities in the context of image classification models. They discovered that common explanations are primarily used due to their simplicity and familiarity while other types of explanation, e.g., domain knowledge, global, textual, active, interactive, and binary explanations are still useful to achieve a variety of objectives.…”

Gesticulate for Health’s Sake! Understanding the Use of Gestures as an Input Modality for Microtask Crowdsourcing

“…The Classification microtasks (Figure 3) ask workers to classify the beak shape of a bird in an image using a set of example images for 8 different beak shapes. A series of 10 images are shown, sampled from the dataset originally produced by Balayn et al (2022). For each image, the worker is asked to select the beak type from the candidates that matches the beak type in the image.…”