Intuitively Assessing ML Model Reliability through Example-Based Explanations and Editing Model Inputs

(Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in Court

“…Example metrics that participants proposed included error rates for determinations that a piece of evidence was inconclusive and ranges for the possible values that false positive software outputs could take on (Section 5.2.3). Together, these findings echo gaps between evaluation design and real-world use contexts highlighted in prior work (e.g., [42,72,81,92]), and, importantly, demonstrate the valuable insights that public defenders develop through their everyday encounters with CFS in the U.S. criminal legal system, further motivating growing efforts in HCI to engage downstream stakeholders in designing performance evaluations of AI systems (e.g., [28,55,76,81]).…”

“…For instance, test inputs may not be sufficiently representative of real-world settings [53,72], and performance metrics may not align with users' preferences and perceptions of ideal model performance [47,53,66]. To address this gap, a growing body of work in HCI aims to design performance evaluations grounded in downstream deployment contexts and the needs and goals of downstream stakeholders (e.g., [18,57,80,81]). This typically involves exploring users' domain-specific information needs [19,46], directly working with downstream stakeholders to collaboratively design evaluation datasets and metrics [80], and designing tools that allow users to specify their own test datasets and performance metrics [18,27,28,55,81].…”

“…To address this gap, a growing body of work in HCI aims to design performance evaluations grounded in downstream deployment contexts and the needs and goals of downstream stakeholders (e.g., [18,57,80,81]). This typically involves exploring users' domain-specific information needs [19,46], directly working with downstream stakeholders to collaboratively design evaluation datasets and metrics [80], and designing tools that allow users to specify their own test datasets and performance metrics [18,27,28,55,81]. This "participatory turn" [26] in the design of performance evaluations highlights the importance and strength of centering the experiential and domain expertise of stakeholders in downstream deployment contexts.…”

“…6.2.1 Uncover and address barriers to engaging frontline workers in evaluation design. Recent work in user-engaged auditing and contextualized evaluation highlights the crucial insights that groups with experiential and domain knowledge (e.g., everyday users and domain experts) can bring to the design of performance evaluations [27,81]. While public defenders possess crucial experiential expertise that can ensure more rigorous testing of CFS reliability, our participants illustrated how insufficient documentation and disclosure practices may result in public defenders seeing only a subset of instances, or a subset of details about instances, in which the government uses CFS (Sections 5.1.1, 5.1.2).…”

Are Metrics Enough? Guidelines for Communicating and Visualizing Predictive Models to Subject Matter Experts

“…Of course, there is an indisputable trade-off cost to building interactive tools and interfaces for model performance. There is continued research being done to support effective and accessible interaction for web-based tools [55], computational notebooks [62], and visualization tools [58] that are practical for data scientists to use. Adding interaction also introduces costs for the audience (e.g., false discoveries) during a presentation [38].…”