ProtoSteer: Steering Deep Sequence Model with Prototypes

Visual analytics systems integrate interactive visualizations and machine learning to enable expert users to solve complex analysis tasks. Applications combine techniques from various fields of research and are consequently not trivial to evaluate. The result is a lack of structure and comparability between evaluations. In this survey, we provide a comprehensive overview of evaluations in the field of human‐centered machine learning. We particularly focus on human‐related factors that influence trust, interpretability, and explainability. We analyze the evaluations presented in papers from top conferences and journals in information visualization and human‐computer interaction to provide a systematic review of their setup and findings. From this survey, we distill design dimensions for structured evaluations, identify evaluation gaps, and derive future research opportunities.

Section: Evaluating the Technique Contributions Of Hcmlmentioning

confidence: 99%

“…The majority of studies (62%) reported values for this dimension detailing expertise levels (e.g. [KPN16; LSL∗17; MLMP18; MXC∗20]) and distribution across levels (e.g., [CVL∗18; SMD∗16]). Few papers use this dimension as controlled study condition, comparing results across participants' expertise levels (e.g., [ESKC18]).…”

Section: Dimensions Of Analysismentioning

confidence: 99%

A Survey of Human‐Centered Evaluations in Human‐Centered Machine Learning

Sperrle

El‐Assady

Guo

et al. 2021

“…Visual comparison approaches have been developed for many model types. For example, approaches exist for specialized types of models, such as sequence models [MXC*20, SGB*19], or for specific data types, such as set data [AMA*14]. Other approaches focus on clustering [KEV*17, CD19], topic models [AG16], word vector embeddings [HG18], and climate models [DWOB20].…”

Section: Related Workmentioning

confidence: 99%

Boxer: Interactive Comparison of Classifier Results

Gleicher

Barve

et al. 2020

Machine learning practitioners often compare the results of different classifiers to help select, diagnose and tune models. We present Boxer, a system to enable such comparison. Our system facilitates interactive exploration of the experimental results obtained by applying multiple classifiers to a common set of model inputs. The approach focuses on allowing the user to identify interesting subsets of training and testing instances and comparing performance of the classifiers on these subsets. The system couples standard visual designs with set algebra interactions and comparative elements. This allows the user to compose and coordinate views to specify subsets and assess classifier performance on them. The flexibility of these compositions allow the user to address a wide range of scenarios in developing and assessing classifiers. We demonstrate Boxer in use cases including model selection, tuning, fairness assessment, and data quality diagnosis.

“…With regard to dimensionality of the visual display, almost all visualizations (196) are 2D , such as [dBD*12, JJ09, MXC*20, SDMT16]. An exception is the interactive visualization technique by Coimbra et al [CMN*16] that adapts and improves biplots to show the data attributes in the projected three‐dimensional (3D) space.…”

Section: In‐depth Categorization Of Trust Against Facets Of Interamentioning

confidence: 99%

The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations

Chatzimparmpas

Martins

Jusufi

et al. 2020

144

Machine learning (ML) models are nowadays used in complex applications in various domains, such as medicine, bioinformatics, and other sciences. Due to their black box nature, however, it may sometimes be hard to understand and trust the results they provide. This has increased the demand for reliable visualization tools related to enhancing trust in ML models, which has become a prominent topic of research in the visualization community over the past decades. To provide an overview and present the frontiers of current research on the topic, we present a State‐of‐the‐Art Report (STAR) on enhancing trust in ML models with the use of interactive visualization. We define and describe the background of the topic, introduce a categorization for visualization techniques that aim to accomplish this goal, and discuss insights and opportunities for future research directions. Among our contributions is a categorization of trust against different facets of interactive ML, expanded and improved from previous research. Our results are investigated from different analytical perspectives: (a) providing a statistical overview, (b) summarizing key findings, (c) performing topic analyses, and (d) exploring the data sets used in the individual papers, all with the support of an interactive web‐based survey browser. We intend this survey to be beneficial for visualization researchers whose interests involve making ML models more trustworthy, as well as researchers and practitioners from other disciplines in their search for effective visualization techniques suitable for solving their tasks with confidence and conveying meaning to their data.