Learning about AI ethics from cases: A scoping review of AI incident repositories and cases

Abstract

Background Cases provide a practical resource for learning regarding the uses and challenges of AI applications. Cases give insight into how principles and values are implicated in real contexts, the trade-offs and different perspectives held regarding these contexts, and the – sometimes hidden – relationships between cases, relationships that may support analogical reasoning across contexts. Objective We aim to (1) provide an approach for structuring ethics cases and (2) investigate how repositories of cases structure those cases and their content. Approach : We motivate a scoping review through a conceptual analysis of ethics case desiderata. The review sought to retrieve repositories, (sometimes known as observatories, catalogues, galleries, or incident databases), and their cases, for analysis of their expression of ethics concepts. Results We identify n = 14 repositories, extracting the case schema used in each, to identify how this metadata can express ethical concepts. We find that most repositories focus on harm-indicators, with some indicating positive impacts, but with little explicit reference to ethical concepts; a subset (n = 4) includes no structural elements addressing ethical concepts or impacts. We extract a subset of cases from the total cases (n = 2,000) across repositories, identifying and extracting the subset of cases addressing education (n = 100). These are grouped by topic, with a structured content analysis provided of ethical implications from one sub-theme, offering qualitative insights into the ethical coverage. Implications Our conceptual analysis and empirical review exemplify a model for ethics cases (shorthanded as Ethics-case-CPR), while highlighting gaps both in existing case repositories and specific examples of cases.