NLIZE: A Perturbation-Driven Visual Interrogation Tool for Analyzing and Interpreting Natural Language Inference Models

Liu, Shusen; Li, Zhimin; Li, Tao; Srikumar, Vivek; Pascucci, Valerio; Bremer, Peer-Timo

doi:10.1109/tvcg.2018.2865230

Cited by 42 publications

(32 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The data type(s) used in each system is another relevant dimension for comparing human‐centered evaluations. The predominant data types are multivariate data (e.g., [PNKC20; WMJ∗19; XMT∗20]); text (e.g., [ARO∗17; ESD∗19; LLL∗19]); and images (e.g., [CRH∗19; LSL∗17; SSSE20]). Only very few papers use other data types like videos (e.g., [KAY∗19]) or geo data (e.g., [PZDD19]).…”

Section: Dimensions Of Analysismentioning

confidence: 99%

“…Explore – Exploration is the task most frequently referenced in evaluation descriptions. Many papers primarily focused on case studies (e.g, [XXM∗19]), use cases (e.g., [CMQ20; LLL∗19]) and questionnaires (e.g., [WSW∗18]). Stahnke et al introduced probing as “ a general interaction approach for information visualization that is aimed at both exploring the data as well as examining its representation ” [SDMT16].…”

Section: Evaluating the Technique Contributions Of Hcmlmentioning

confidence: 99%

See 1 more Smart Citation

A Survey of Human‐Centered Evaluations in Human‐Centered Machine Learning

Sperrle

El‐Assady

Guo

et al. 2021

Computer Graphics Forum

View full text Add to dashboard Cite

Visual analytics systems integrate interactive visualizations and machine learning to enable expert users to solve complex analysis tasks. Applications combine techniques from various fields of research and are consequently not trivial to evaluate. The result is a lack of structure and comparability between evaluations. In this survey, we provide a comprehensive overview of evaluations in the field of human‐centered machine learning. We particularly focus on human‐related factors that influence trust, interpretability, and explainability. We analyze the evaluations presented in papers from top conferences and journals in information visualization and human‐computer interaction to provide a systematic review of their setup and findings. From this survey, we distill design dimensions for structured evaluations, identify evaluation gaps, and derive future research opportunities.

show abstract

Section: Dimensions Of Analysismentioning

confidence: 99%

Section: Evaluating the Technique Contributions Of Hcmlmentioning

confidence: 99%

A Survey of Human‐Centered Evaluations in Human‐Centered Machine Learning

Sperrle

El‐Assady

Guo

et al. 2021

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…Ultimately, from the further analysis of the statistics, we conclude that in five cases both domain and ML experts used visualization tools and evaluated them. In 32 cases, e.g., [CS14, KCK*19, LLL*19, MvW11, XYC*18], only domain experts were asked; and in 19 cases only ML experts participated, such as in [LSC*18, NHP*18].…”

Section: In‐depth Categorization Of Trust Against Facets Of Interamentioning

confidence: 99%

The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations

Chatzimparmpas

Martins

Jusufi

et al. 2020

Computer Graphics Forum

143

View full text Add to dashboard Cite

Machine learning (ML) models are nowadays used in complex applications in various domains, such as medicine, bioinformatics, and other sciences. Due to their black box nature, however, it may sometimes be hard to understand and trust the results they provide. This has increased the demand for reliable visualization tools related to enhancing trust in ML models, which has become a prominent topic of research in the visualization community over the past decades. To provide an overview and present the frontiers of current research on the topic, we present a State‐of‐the‐Art Report (STAR) on enhancing trust in ML models with the use of interactive visualization. We define and describe the background of the topic, introduce a categorization for visualization techniques that aim to accomplish this goal, and discuss insights and opportunities for future research directions. Among our contributions is a categorization of trust against different facets of interactive ML, expanded and improved from previous research. Our results are investigated from different analytical perspectives: (a) providing a statistical overview, (b) summarizing key findings, (c) performing topic analyses, and (d) exploring the data sets used in the individual papers, all with the support of an interactive web‐based survey browser. We intend this survey to be beneficial for visualization researchers whose interests involve making ML models more trustworthy, as well as researchers and practitioners from other disciplines in their search for effective visualization techniques suitable for solving their tasks with confidence and conveying meaning to their data.

show abstract

“…The multi-level matching attention of our work is closely related to the issue of sequences matching. The task of sequences matching aims to compare two sequences and identify the relationship between them, such as paraphrase identification [32], natural language inference [33] and answer sentence selection [34]. In these domains, without fine-grained interaction, learning the representation of each sequence independently has been proven to be outperformed by that of attention mechanism with sophisticated interaction.…”

Section: Sequence Matching and Interactionmentioning

confidence: 99%

Multi-Level Fine-Grained Interactions for Collaborative Filtering

Feng

Zeng

2019

IEEE Access

View full text Add to dashboard Cite

In recent years, review-based collaborative filtering (CF) has been extensively studied, which is an combination between natural language processing (NLP) and recommender systems. The core pattern behind CF is to first model user and item, and then adopts a relatively primitive interaction between them for personalized recommendation. This pattern is very similar to the issue of sequence matching in NLP, where sequence 1 and sequence 2 are matched with a fine-grained interaction leading to a better result. Therefore, there is a tremendous room for further improvement in current review-based CF to release the power of fine-grained interaction. To this end, we treat the user review set and item review set as two sequences, and design a multi-level matching attention layer for fine-grained interaction. In addition, we devise the aspect-level and review-level attention to measure the contribution of each review. Extensive experiments on 24 public datasets show that the proposed model consistently outperforms the state-of-theart approaches. More importantly, by selecting the relevant reviews according to the aspect attention score and review attention score, we can observe which specific item aspects that user mainly concerned and which item characteristic highly matched with the user preference, in which the recommendation interpretability can be enhanced.

show abstract

NLIZE: A Perturbation-Driven Visual Interrogation Tool for Analyzing and Interpreting Natural Language Inference Models

Cited by 42 publications

References 19 publications

A Survey of Human‐Centered Evaluations in Human‐Centered Machine Learning

A Survey of Human‐Centered Evaluations in Human‐Centered Machine Learning

The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations

Multi-Level Fine-Grained Interactions for Collaborative Filtering

Contact Info

Product

Resources

About