ProactiV: Studying deep learning model behavior under input transformations

Abstract

<p>Deep learning (DL) models have shown performance benefits across many applications, from classification to, more recently, image-to-image translation. However, low interpretability often leads to unexpected model behavior when deployed in the real world. Usually, this unexpected behavior is because the training data domain does not reflect the deployment data domain. Identifying model breaking points under input conditions and domain shifts, i.e., input transformations, is essential to improve models. Although visual analytics (VA) has shown promise in studying the behavior of model outputs under continually varying inputs, existing methods mainly focus on per class or instance-level analysis. We aim to generalize beyond classification where classes do not exist and provide a global holistic model behavior overview under co-occurring input transformations. We present a DL model-agnostic VA method (ProactiV ) to help model developers proactively study output behavior under input transformations to identify and verify breakpoints. ProactiV relies on a proposed input optimization method to determine how far a given transformed input has to move in the input data manifold to achieve the desired output. The data from this optimization process allows the study of global and local model output behavior under input transformations at scale. Additionally, the optimization method provides insight into the input characteristics that result in desired outputs and helps recognize model biases. We highlight how ProactiV effectively supports studying model behavior under input transformations to identify and verify model breakpoints with example classification and image-to-image translation tasks.</p>

The Transform-and-Perform framework: Explainable deep learning beyond classification

“…6f, most object modes emerge earlier in the generation process, primarily within the inner rings. This indicates that the model generates most global structures within the first few iterations, a finding supported by literature [6,26,33,39]. Next, we dive deeper into the evolution of finer features to enhance our understanding of the generation process.…”

“…For instance, the evolution of shapes [47], internal model, i.e., Unet features [7], or the evolution of concept-keyword associations over iterations [25] across various datasets could be explored. Exploring the Unet features is not straightforward since they do not have a latent space, owing to their skip connections, and different layers are important in different iteration steps [33,39]. Finally, developing advanced interactive interfaces could enhance the analysis of the evolutionary embedding layout generated by T DL.…”

“…However, more effective interactive sampling methods can be used. For example, literature [6,33] indicates a composition phase where content is created, followed by a denoising phase to refine details. Users could interactively adjust the sampling, potentially selecting more steps initially and fewer later in the process to facilitate more efficient analysis.…”

“…Concerning the generation process, literature has focused on exploring the purpose of iterations, revealing that different stages in the denoising steps correspond to varying granularity of visual concepts [6,25,33,39]. However, it is crucial to understand the evolution of features or dataset modes to understand the root causes of failures in these models and enable downstream development [19,25].…”

“…It sheds light on the feature evolution and the decisionmaking dynamics of dataset modes. Such knowledge is relevant for refining model architectures and training strategies [6,14,51], controlling the generation process [26,49], and improving overall model performance [33,39]. This analysis could also help identify the biases and attribute entanglements relevant for robust model development [51].…”