Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers

Latent spaces—reduced‐dimensionality vector space embeddings of data, fit via machine learning—have been shown to capture interesting semantic properties and support data analysis and synthesis within a domain. Interpretation of latent spaces is challenging because prior knowledge, sometimes subtle and implicit, is essential to the process. We contribute methods for “latent space cartography”, the process of mapping and comparing meaningful semantic dimensions within latent spaces. We first perform a literature survey of relevant machine learning, natural language processing, and scientific research to distill common tasks and propose a workflow process. Next, we present an integrated visual analysis system for supporting this workflow, enabling users to discover, define, and verify meaningful relationships among data points, encoded within latent space dimensions. Three case studies demonstrate how users of our system can compare latent space variants in image generation, challenge existing findings on cancer transcriptomes, and assess a word embedding benchmark.

Section: Related Workmentioning

confidence: 99%

Latent Space Cartography: Visual Analysis of Vector Space Embeddings

Liu

Jun

et al. 2019

“…Systems have been built to provide insight into how a machine learning model makes predictions by highlighting individual data instances that the model predicts poorly. With Squares [RAL∗17], analysts can view classification models based on an in‐depth analysis of label distribution on a test data set. Krause et.…”

Section: Related Workmentioning

confidence: 99%

“…Users want to judge model predictions in coordination with exploratory data analysis views. Model explanation techniques such as those linking confusion matrix cells to individual data instances offer a good example of tight linking between the data space and the model space [ZWM∗18,ACD∗15,RAL∗17]. G6: Transition seamlessly from one step to another in the overall workflow : Providing a seamless workflow in the resulting interface helps the user to perform the different tasks required in generating and selecting the relevant models.…”

Section: A Workflow For Exploratory Model Analysismentioning

confidence: 99%

A User‐based Visual Analytics Workflow for Exploratory Model Analysis

Cashman

Humayoun

Heimerl

et al. 2019

Many visual analytics systems allow users to interact with machine learning models towards the goals of data exploration and insight generation on a given dataset. However, in some situations, insights may be less important than the production of an accurate predictive model for future use. In that case, users are more interested in generating of diverse and robust predictive models, verifying their performance on holdout data, and selecting the most suitable model for their usage scenario. In this paper, we consider the concept of Exploratory Model Analysis (EMA), which is defined as the process of discovering and selecting relevant models that can be used to make predictions on a data source. We delineate the differences between EMA and the well‐known term exploratory data analysis in terms of the desired outcome of the analytic process: insights into the data or a set of deployable models. The contributions of this work are a visual analytics system workflow for EMA, a user study, and two use cases validating the effectiveness of the workflow. We found that our system workflow enabled users to generate complex models, to assess them for various qualities, and to select the most relevant model for their task.

“…Work has been done on performance analysis of predictive models from a general perspective, focusing on the visual exploration of several performance indicators. ModelTracker [ACD∗15] and Squares [RAL∗17] are two systems that intend to provide insights into the performance of classifiers. Although they share a common goal, their approaches differ.…”

Section: Related Workmentioning

confidence: 99%

V‐Awake: A Visual Analytics Approach for Correcting Sleep Predictions from Deep Learning Models

Caballero

Westenberg

Gebre

et al. 2019

The usage of deep learning models for tagging input data has increased over the past years because of their accuracy and high‐performance. A successful application is to score sleep stages. In this scenario, models are trained to predict the sleep stages of individuals. Although their predictive accuracy is high, there are still mis classifications that prevent doctors from properly diagnosing sleep‐related disorders. This paper presents a system that allows users to explore the output of deep learning models in a real‐life scenario to spot and analyze faulty predictions. These can be corrected by users to generate a sequence of sleep stages to be examined by doctors. Our approach addresses a real‐life scenario with absence of ground truth. It differs from others in that our goal is not to improve the model itself, but to correct the predictions it provides. We demonstrate that our approach is effective in identifying faulty predictions and helping users to fix them in the proposed use case.