FDHelper: Assist Unsupervised Fraud Detection Experts with Interactive Feature Selection and Evaluation

A Survey of Human‐Centered Evaluations in Human‐Centered Machine Learning

“…We found a good distribution of both direct and indirect manipulation in the papers we surveyed, with most systems using only of both types. Direct manipulation is often used for selecting and filtering data instances (e.g., [CYL∗20; KAKC18; KBJ∗20; MXLM20; WGYS18]), while indirect manipulation techniques tend to be used for data labeling (e.g., [BHZ∗18; XXM∗19]) or model specification and hyperparameter tuning (e.g., [CHH∗19; DLW∗17; JSR∗19; KEV∗18; PNKC20; SLC∗20; XXM∗19]). These tasks are not exclusive to each interaction manipulation technique.…”

“…While papers describe interactions at varying levels of detail, a few common interaction types emerged in our analysis, including selection (e.g., [KBJ∗20; WBL∗20]), filtering (e.g., [LGM∗20; PNKC20]), zooming (e.g., [LPH∗20; LSC∗18]), tuning weights/hyperparameters (e.g., [DSKE20; SLC∗20]), and annotation/labeling (e.g., [SSKE19; XXM∗19]). Multiple of these interaction types tend to be combined in a single system, for example when users first select a set of data instances before labeling them (e.g., [BHZ∗18; ZWLC19]).…”

“…However, only 14% of papers in this survey measured whether interpretability was achieved. Studies that evaluate interpretability often test how well people can communicate their internalized understanding of how models make decisions (e.g., [HHC∗19; KAS∗20; SLC∗20]), often through qualitative responses from participants.…”

“…Both papers reported qualitative responses from participants [DSKE20; WGSY19]. The ability for domain experts to control and correct model outcomes or processes is often motivated by use cases where the decisions being made are critical and can have detrimental outcomes if not seriously considered (e.g., healthcare [CRH∗19] and fraud detection [SLC∗20]).…”

“…The motivation of human‐centered aspects that are not explicitly evaluated is a common pattern that appears in many papers. Sun et al [SLC∗20] evaluate their system for fraud detection with both expert and novice users. An expert interview reveals that experts have higher confidence in their diagnosis when using the system but does not elaborate on the baseline.…”