No abstract

Artificial intelligence (AI) solutions that automatically extract information from digital histology images have shown great promise for improving pathological diagnosis. Prior to routine use, it is important to evaluate their predictive performance and obtain regulatory approval. This assessment requires appropriate test datasets. However, compiling such datasets is challenging and specific recommendations are missing. A committee of various stakeholders, including commercial AI developers, pathologists, and researchers, discussed key aspects and conducted extensive literature reviews on test datasets in pathology. Here, we summarize the results and derive general recommendations on compiling test datasets. We address several questions: Which and how many images are needed? How to deal with low-prevalence subsets? How can potential bias be detected? How should datasets be reported? What are the regulatory requirements in different countries? The recommendations are intended to help AI developers demonstrate the utility of their products and to help pathologists and regulatory agencies verify reported performance measures. Further research is needed to formulate criteria for sufficiently representative test datasets so that AI solutions can operate with less user intervention and better support diagnostic workflows in the future.

The current move towards digital pathology enables pathologists to use artificial intelligence (AI)‐based computer programmes for the advanced analysis of whole slide images. However, currently, the best‐performing AI algorithms for image analysis are deemed black boxes since it remains – even to their developers – often unclear why the algorithm delivered a particular result. Especially in medicine, a better understanding of algorithmic decisions is essential to avoid mistakes and adverse effects on patients. This review article aims to provide medical experts with insights on the issue of explainability in digital pathology. A short introduction to the relevant underlying core concepts of machine learning shall nurture the reader's understanding of why explainability is a specific issue in this field. Addressing this issue of explainability, the rapidly evolving research field of explainable AI (XAI) has developed many techniques and methods to make black‐box machine‐learning systems more transparent. These XAI methods are a first step towards making black‐box AI systems understandable by humans. However, we argue that an explanation interface must complement these explainable models to make their results useful to human stakeholders and achieve a high level of causability, i.e. a high level of causal understanding by the user. This is especially relevant in the medical field since explainability and causability play a crucial role also for compliance with regulatory requirements. We conclude by promoting the need for novel user interfaces for AI applications in pathology, which enable contextual understanding and allow the medical expert to ask interactive ‘what‐if’‐questions. In pathology, such user interfaces will not only be important to achieve a high level of causability. They will also be crucial for keeping the human‐in‐the‐loop and bringing medical experts' experience and conceptual knowledge to AI processes.

Holzinger et al.: Personas for AI: An Open Source ToolboxThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.

Background: Artificial Intelligence (AI) is becoming more and more important especially in datacentric fields, such as biomedical research and biobanking. However, AI does not only offer advantages and promising benefits, but brings about also ethical risks and perils. In recent years, there has been growing interest in AI ethics, as reflected by a huge number of (scientific) literature dealing with the topic of AI ethics. The main objectives of this review are: (1) to provide an overview about important (upcoming) AI ethics regulations and international recommendations as well as available AI ethics tools and frameworks relevant to biomedical research, (2) to identify what AI ethics can learn from findings in ethics of traditional biomedical research - in particular looking at ethics in the domain of biobanking, and (3) to provide an overview about the main research questions in the field of AI ethics in biomedical research. Methods: We adopted a modified thematic review approach focused on understanding AI ethics aspects relevant to biomedical research. For this review, four scientific literature databases at the cross-section of medical, technical, and ethics science literature were queried: PubMed, BMC Medical Ethics, IEEE Xplore, and Google Scholar. In addition, a grey literature search was conducted to identify current trends in legislation and standardization. Results: More than 2,500 potentially relevant publications were retrieved through the initial search and 57 documents were included in the final review. The review found many documents describing high-level principles of AI ethics, and some publications describing approaches for making AI ethics more actionable and bridging the principles-to-practice gap. Also, some ongoing regulatory and standardization initiatives related to AI ethics were identified. It was found that ethical aspects of AI implementation in biobanks are often like those in biomedical research, for example with regards to handling big data or tackling informed consent. The review revealed current ‘hot’ topics in AI ethics related to biomedical research. Furthermore, several published tools and methods aiming to support practical implementation of AI ethics, as well as tools and frameworks specifically addressing complete and transparent reporting of biomedical studies involving AI are described in the review results. Conclusions: The review results provide a practically useful overview of research strands as well as regulations, guidelines, and tools regarding AI ethics in biomedical research. Furthermore, the review results show the need for an ethical-mindful and balanced approach to AI in biomedical research, and specifically reveal the need for AI ethics research focused on understanding and resolving practical problems arising from the use of AI in science and society.