Evaluating the Effect of Feedback from Different Computer Vision Processing Stages

Kittley-Davies, Jacob; Alqaraawi, Ahmed; Yang, Rayoung; Costanza, Enrico; Rogers, Alex; Stein, Sebastian

doi:10.1145/3290605.3300273

Cited by 7 publications

(11 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multiple of these interaction types tend to be combined in a single system, for example when users first select a set of data instances before labeling them (e.g., [BHZ∗18; ZWLC19]). Less frequently, we also observed more complex combinations of interactions to enable application‐specific operations such as cropping an image to the area of interest [CRH∗19], aligning video frames [KAY∗19], or placing game level elements[GLC∗19]. In several cases, these are observed by systems and used as implicit feedback towards an underlying model.…”

Section: Dimensions Of Analysismentioning

confidence: 99%

“…Video Analysis – Kittley et al [KAY∗19] aimed to investigate the intelligibility of different processing steps throughout the computer vision pipeline. They task study participants of their controlled lab study with designing a stop‐motion video, as this task can be supported through explainable keyframe matching and is “ not obvious to participants ” [KAY∗19]. Each participant completes four tasks that are designed to showcase the system (1), reveal algorithm failure conditions (2), and assess user understanding (3+4).…”

Section: Application‐specific Evaluationsmentioning

confidence: 99%

See 1 more Smart Citation

A Survey of Human‐Centered Evaluations in Human‐Centered Machine Learning

Sperrle

El‐Assady

Guo

et al. 2021

Computer Graphics Forum

View full text Add to dashboard Cite

Visual analytics systems integrate interactive visualizations and machine learning to enable expert users to solve complex analysis tasks. Applications combine techniques from various fields of research and are consequently not trivial to evaluate. The result is a lack of structure and comparability between evaluations. In this survey, we provide a comprehensive overview of evaluations in the field of human‐centered machine learning. We particularly focus on human‐related factors that influence trust, interpretability, and explainability. We analyze the evaluations presented in papers from top conferences and journals in information visualization and human‐computer interaction to provide a systematic review of their setup and findings. From this survey, we distill design dimensions for structured evaluations, identify evaluation gaps, and derive future research opportunities.

show abstract

Section: Dimensions Of Analysismentioning

confidence: 99%

Section: Application‐specific Evaluationsmentioning

confidence: 99%

A Survey of Human‐Centered Evaluations in Human‐Centered Machine Learning

Sperrle

El‐Assady

Guo

et al. 2021

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…Saliency map explanations have mostly been evaluated with simulation metrics and rarely with human subjects [5,46].…”

Section: Sensitive Measures and Tasks For Evaluating Explanation Fait...mentioning

confidence: 99%

“…and the efficacy of solutions (problem no longer perceivable or perceptually forgivable?). However, designing successful experiments with strong effects and sensitive measures is difficult and many studies fail to find effects [9,46,73]. To improve the sensitivity, experiments need more sensitive measures and carefully designed participant tasks.…”

Section: Sensitive Measures and Tasks For Evaluating Explanation Fait...mentioning

confidence: 99%

Debiased-CAM to mitigate systematic error with faithful visual explanations of machine learning

Zhang¹,

Dimiccoli²,

Lim³

2022

Preprint

View full text Add to dashboard Cite

Model explanations such as saliency maps can improve user trust in AI by highlighting important features for a prediction. However, these become distorted and misleading when explaining predictions of images that are subject to systematic error (bias). Furthermore, the distortions persist despite model fine-tuning on images biased by different factors (blur, color temperature, day/night). We present Debiased-CAM to recover explanation faithfulness across various bias types and levels by training a multi-input, multi-task model with auxiliary tasks for explanation and bias level predictions. In simulation studies, the approach not only enhanced prediction accuracy, but also generated highly faithful explanations about these predictions as if the images were unbiased. In user studies, debiased explanations improved user task performance, perceived truthfulness and perceived helpfulness. Debiased training can provide a versatile platform for robust performance and explanation faithfulness for a wide range of applications with data biases.

show abstract

“…Despite these existing applications, how to effectively communicate the results of computer vision systems to end-users is an open research question in HCI [28]. Work on intelligent image overlays in the medical domain has predominantly focused on still imaging, with radiology being a key area of development [23].…”

Section: Video Annotationmentioning

confidence: 99%

Designing Visual Markers for Continuous Artificial Intelligence Support

Berkel

Ahmad

Stoyanov

et al. 2020

ACM Trans. Comput. Healthcare

View full text Add to dashboard Cite

Colonoscopy, the visual inspection of the large bowel using an endoscope, offers protection against colorectal cancer by allowing for the detection and removal of pre-cancerous polyps. The literature on polyp detection shows widely varying miss rates among clinicians, with averages ranging around 22%--27%. While recent work has considered the use of AI support systems for polyp detection, how to visualise and integrate these systems into clinical practice is an open question. In this work, we explore the design of visual markers as used in an AI support system for colonoscopy. Supported by the gastroenterologists in our team, we designed seven unique visual markers and rendered them on real-life patient video footage. Through an online survey targeting relevant clinical staff ( N = 36), we evaluated these designs and obtained initial insights and understanding into the way in which clinical staff envision AI to integrate in their daily work-environment. Our results provide concrete recommendations for the future deployment of AI support systems in continuous, adaptive scenarios.

show abstract

Evaluating the Effect of Feedback from Different Computer Vision Processing Stages

Cited by 7 publications

References 28 publications

A Survey of Human‐Centered Evaluations in Human‐Centered Machine Learning

A Survey of Human‐Centered Evaluations in Human‐Centered Machine Learning

Debiased-CAM to mitigate systematic error with faithful visual explanations of machine learning

Designing Visual Markers for Continuous Artificial Intelligence Support

Contact Info

Product

Resources

About