Pilot study to evaluate tools to collect pathologist annotations for validating machine learning algorithms

Elfer, Katherine; Dudgeon, Sarah; Garcia, Victor; Blenman, Kim; Hytopoulos, Evangelos; Wen, Si; Li, Xiaoxian; Ly, Amy; Werness, Bruce; Sheth, Manasi; Amgad, Mohamed; Gupta, Rajarsi; Saltz, Joel; Hanna, Matthew G.; Ehinger, Anna; Peeters, Dieter; Salgado, Roberto; Gallas, Brandon D.

doi:10.1117/1.jmi.9.4.047501

Cited by 5 publications

(6 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…eeDAP is a software-hardware interface to register microscope slides to WSI images, collect pathologist annotations, and evaluate WSI technology. Prior registration accuracy studies 14 and pathologist annotation studies 17 have demonstrated the accuracy, stability, and utility of eeDAP as a data-collection tool. This work shows that with new hardware eeDAP is still capable of registration accuracy within 5 µm on a glass slide 97% of the time.…”

Section: Discussionmentioning

confidence: 99%

“…The new hardware was motivated by a pilot study of the High Throughput Truthing (HTT) project [15][16] . Results from the pilot study demonstrated the need to improve overall speed of eeDAP to make its use comparable to that of our web application data collection platforms 17 . We combine the data collected on the new hardware with the 2018 data and perform a multi-reader multi-case (MRMC) analysis to obtain estimates of reader and case variance components for sizing future studies.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Analysis of a pilot study collecting pathologist annotations for validating machine learning algorithms

Elfer

Garcia

Gallas

2023

Medical Imaging 2023: Image Perception, Observer Performance, and Technology Assessment

View full text Add to dashboard Cite

We describe registration accuracy studies of a custom hardware-software system called eeDAP that registers fields of view (FOVs) of a glass slide on a microscope to the digital presentations of regions of interest (ROIs) of whole slide images (WSI) of the same glass slide. In this manuscript, we describe the results of adding new hardware and use the results to size a larger pathologist data collection study. We create a registration accuracy task by identifying a visually distinct target. This target will be the center of a WSI ROI and is expected to appear in the center of the microscope FOV. We examined the registration accuracy of 60 ROIs from six slides, alternating registration methods and slide order within each study. We measure the distance between the target and the FOV center (registration error) using an eye piece reticle ruler as the stage moves from target to target. We summarize each error as a success (≤ 5.0 µm) or failure (> 5.0 µm). We completed a multi-reader multicase (MRMC) analysis of the registration successes and failures to estimate the variance components due to the readers and the cases. When using eeDAP in-focus, accuracy was within 5 µm in more than 97% of the FOVs. The eeDAP registration methods were robust to new hardware, and the MRMC analysis has provided variance components for sizing future registration accuracy studies to account for the variability from readers and cases.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Analysis of a pilot study collecting pathologist annotations for validating machine learning algorithms

Elfer

Garcia

Gallas

2023

Medical Imaging 2023: Image Perception, Observer Performance, and Technology Assessment

View full text Add to dashboard Cite

show abstract

“…New tools are emerging for estimating truth from panels of expert readers, including improved approaches to reader training to reduce reader variability, improved collection interfaces, and statistical analysis tools. 36,37 Similarly, better approaches are emerging for collecting data from humans who are serving as "study" readers in a task-based assessment. These also include improved training methods and data collection interfaces that facilitate the collection of more precise reader data on a finer measurement scale.…”

Section: Methodsmentioning

confidence: 99%

State of the art in the task-based assessment of medical imaging systems

Myers¹

2023

Medical Imaging 2023: Physics of Medical Imaging

View full text Add to dashboard Cite

Image Science provides a framework for the task-based assessment of image quality. This framework has been used to support the evaluation of medical imaging system hardware, iterative reconstruction algorithms and other image processing methods, and display devices by academia, industry, and US FDA. Since the earliest instances of the SPIE Medical Imaging Symposium the conference has served as an essential venue for presentations and discussions related to the objective assessment of image quality, featuring first disclosures of new models for physiological backgrounds and pathologies, tools for simulating medical imaging systems, models for the human and Bayesian observer, and methods for computing task-based figures of merit. This paper highlights recent advances in the objective or task-based assessment of image quality through the use of computational models and methods, and points to new initiatives intended to develop resources to move this important field forward.

show abstract

“…Pathologists' visual assessment of TILs in biopsies and surgical resections of human epidermal growth factor receptor-2 positive (HER2+) and triple-negative breast cancer (TNBC) patients results in a (TILs) score ranging from 0 to 100 [2]. However, there is a great deal of variability among pathologists in estimating the TILs score due to the visualdependent nature of the estimation task [3][4][5]. Advances in ML algorithms in digital pathology [6] pave the way for designing algorithms to automatically generate TILs scores from whole slide images (WSIs) of breast cancer patients.…”

Section: Introductionmentioning

confidence: 99%

Assessment of machine learning algorithms for TILs scoring using whole slide images: comparison with pathologists

Arab,

Garcia,

M. Kahaki

et al. 2024

Medical Imaging 2024: Digital and Computational Pathology

View full text Add to dashboard Cite

Machine learning (ML) based whole slide imaging biomarkers have great potential to improve the efficiency and consistency of biomarker quantification, thereby facilitating the development of prognosis models for personalized medicine. Assessment methods in this area are still under-developed. Using the public TiGER (Tumor InfiltratinG lymphocytes in breast cancER) challenge data, we developed a deep neural network-based algorithm for automated tumorinfiltrating lymphocytes (TILs) scoring from whole slide images (WSIs) of biopsies and surgical resections of human epidermal growth factor receptor-2 positive (HER2+) and triple-negative breast cancer (TNBC) patients. The purpose of this study is to assess our model's performance on a new independent dataset. Seven pathologists independently assessed 320 pre-selected regions of interests (ROIs) across 32 WSIs for TILs scoring. Our results show that there is substantial variability among pathologists in scoring TILs density. We also observed a systematic discrepancy between the ML-based TILs scoring and the pathologists' manual scoring that led us to develop a calibration between the two. Our calibration reduced the discrepancy, increasing the intra-class-correlation coefficient (ICC) from 0.35 (95% CI [-0.062, 0.625]) for uncalibrated scores to 0.67 (95% CI [0.6, 0.736]) after calibration.

show abstract

Pilot study to evaluate tools to collect pathologist annotations for validating machine learning algorithms

Cited by 5 publications

References 44 publications

Analysis of a pilot study collecting pathologist annotations for validating machine learning algorithms

Analysis of a pilot study collecting pathologist annotations for validating machine learning algorithms

State of the art in the task-based assessment of medical imaging systems

Assessment of machine learning algorithms for TILs scoring using whole slide images: comparison with pathologists

Contact Info

Product

Resources

About