Development of Training Materials for Pathologists to Provide Machine Learning Validation Data of Tumor-Infiltrating Lymphocytes in Breast Cancer

Abstract

The High Throughput Truthing project aims to develop a dataset for validating artificial intelligence and machine learning models (AI/ML) fit for regulatory purposes. The context of this AI/ML validation dataset is the reporting of stromal tumor-infiltrating lymphocytes (sTILs) density evaluations in hematoxylin and eosin-stained invasive breast cancer biopsy specimens. After completing the pilot study, we found notable variability in the sTILs estimates as well as inconsistencies and gaps in the provided training to pathologists. Using the pilot study data and an expert panel, we created custom training materials to improve pathologist annotation quality for the pivotal study. We categorized regions of interest (ROIs) based on their mean sTILs density and selected ROIs with the highest and lowest sTILs variability. In a series of eight one-hour sessions, the expert panel reviewed each ROI and provided verbal density estimates and comments on features that confounded the sTILs evaluation. We aggregated and shaped the comments to identify pitfalls and instructions to improve our training materials. From these selected ROIs, we created a training set and proficiency test set to improve pathologist training with the goal to improve data collection for the pivotal study. We are not exploring AI/ML performance in this paper. Instead, we are creating materials that will train crowd-sourced pathologists to be the reference standard in a pivotal study to create an AI/ML model validation dataset. The issues discussed here are also important for clinicians to understand about the evaluation of sTILs in clinical practice and can provide insight to developers of AI/ML models.

State of the art in the task-based assessment of medical imaging systems

“…New tools are emerging for estimating truth from panels of expert readers, including improved approaches to reader training to reduce reader variability, improved collection interfaces, and statistical analysis tools. 36,37 Similarly, better approaches are emerging for collecting data from humans who are serving as "study" readers in a task-based assessment. These also include improved training methods and data collection interfaces that facilitate the collection of more precise reader data on a finer measurement scale.…”

Initial interactions with the FDA on developing a validation dataset as a medical device development tool

“…Project members include scientists, clinicians, and the FDA, in collaboration with the Pathology Innovation Collaborative Community [36] and the International Immuno-Oncology Biomarker Working Group [37]. We recently completed a pilot study whereby we collected 64 biopsies of invasive ductal carcinomas, defined 10 ROIs per WSI, and collected 7,373 estimates of sTIL densities from 29 pathologists [9]. Microscopic assessments were enabled by mapping WSI scan coordinates to the corresponding coordinates on the glass slide of the optical microscope using the evaluation environment for digital and analog pathology (eeDAP) system [35].…”

“…The guidelines define a specific biomarker, the density of sTILs, to infer prognostic importance for specific patient cohorts. However, the sTIL assessment is burdensome and fraught with pathologist variability [9,10]. These challenges can be alleviated by computational models powered by artificial intelligence and machine learning (AI/ML) that have had appropriate validation of their bias, variance, and clinical context.…”

Assessment of machine learning algorithms for TILs scoring using whole slide images: comparison with pathologists

“…Pathologists' visual assessment of TILs in biopsies and surgical resections of human epidermal growth factor receptor-2 positive (HER2+) and triple-negative breast cancer (TNBC) patients results in a (TILs) score ranging from 0 to 100 [2]. However, there is a great deal of variability among pathologists in estimating the TILs score due to the visualdependent nature of the estimation task [3][4][5]. Advances in ML algorithms in digital pathology [6] pave the way for designing algorithms to automatically generate TILs scores from whole slide images (WSIs) of breast cancer patients.…”