WeLineation: crowdsourcing delineations for reliable ground truth estimation

Goel, Saksham; Sharma, Yash; Jauer, Malte-Levin; Deserno, Thomas M.

doi:10.1117/12.2551279

Cited by 6 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Experts could then revise these initial annotations. Note that the utility of crowdsourcing in medical image annotation has been demonstrated in multiple studies (Foncubierta-Rodriguez and Muller, 2012;Gurari et al, 2015;Sharma et al, 2017;Goel et al, 2020).…”

Section: Discussion and Future Researchmentioning

confidence: 97%

A Survey on Deep Learning for Skin Lesion Segmentation

Mirikharaji¹,

Barata²,

Kumar³

et al. 2022

Preprint

View full text Add to dashboard Cite

Skin cancer is a major public health problem that could benefit from computer-aided diagnosis to reduce the burden of this common disease. Skin lesion segmentation from images is an important step toward achieving this goal. However, the presence of natural and artificial artifacts (e.g., hair and air bubbles), intrinsic factors (e.g., lesion shape and contrast), and variations in image acquisition conditions make skin lesion segmentation a challenging task. Recently, various researchers have explored the applicability of deep learning models to skin lesion segmentation. In this survey, we cross-examine 134 research papers that deal with deep learning based segmentation of skin lesions. We analyze these works along several dimensions, including input data (datasets, preprocessing, and synthetic data generation), model design (architecture, modules, and losses), and evaluation aspects (data annotation requirements and segmentation performance). We discuss these dimensions both from the viewpoint of select seminal works, and from a systematic viewpoint, examining how those choices have influenced current trends, and how their limitations should be addressed. We summarize all examined works in a comprehensive table to facilitate comparisons.

show abstract

Section: Discussion and Future Researchmentioning

confidence: 97%

A Survey on Deep Learning for Skin Lesion Segmentation

Mirikharaji¹,

Barata²,

Kumar³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Our investigation is based on a dataset that was created using the "WeLineation" crowdsourcing platform for segmentation. 8 Within this study, mostly inexperienced users were performing the segmentations, only instructed by a short written description and one guidance figure (Fig 1). The dataset is composed of 75 photographs of human eyes (Fig.…”

Section: Methodsmentioning

confidence: 99%

“…In previous work, we have established a web-based platform that allows numerous users (the crowd) to provide reference delineation of objects in medical images. 8 In contrast to previous STAPLE investigations, where references were mostly generated by just a few domain experts (one up to three), we examined the results using many ratings per image (30 and more) but generated from mostly inexperienced users.…”

Section: Introductionmentioning

confidence: 99%

STAPLE performance assessed on crowdsourced sclera segmentations

Jauer

Goel

Sharma

et al. 2020

Medical Imaging 2020: Imaging Informatics for Healthcare, Research, and Applications

Self Cite

View full text Add to dashboard Cite

published version features the final layout of the paper including the volume, issue and page numbers. Link to publication General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal.If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the "Taverne" license above, please follow below link for the End User

show abstract

“…Crowd sourcing is a form of subjective consensus reference standard that has been applied to image annotation, image segmentation, and object delineation tasks. 70 It has been shown, in certain settings, that the quality of annotations from experts and those from novices becomes equivalent with an increased number of novices. 71,72 Nevertheless, the use of crowd sourcing as a reference standard for machine-learning applications in medical imaging must be further investigated before it can be recommended for general use.…”

Section: Crowd Sourcingmentioning

confidence: 99%

AAPM task group report 273: Recommendations on best practices for AI and machine learning for computer‐aided diagnosis in medical imaging

et al. 2023

Self Cite

View full text Add to dashboard Cite

Rapid advances in artificial intelligence (AI) and machine learning, and specifically in deep learning (DL) techniques, have enabled broad application of these methods in health care. The promise of the DL approach has spurred further interest in computer‐aided diagnosis (CAD) development and applications using both “traditional” machine learning methods and newer DL‐based methods. We use the term CAD‐AI to refer to this expanded clinical decision support environment that uses traditional and DL‐based AI methods. Numerous studies have been published to date on the development of machine learning tools for computer‐aided, or AI‐assisted, clinical tasks. However, most of these machine learning models are not ready for clinical deployment. It is of paramount importance to ensure that a clinical decision support tool undergoes proper training and rigorous validation of its generalizability and robustness before adoption for patient care in the clinic. To address these important issues, the American Association of Physicists in Medicine (AAPM) Computer‐Aided Image Analysis Subcommittee (CADSC) is charged, in part, to develop recommendations on practices and standards for the development and performance assessment of computer‐aided decision support systems. The committee has previously published two opinion papers on the evaluation of CAD systems and issues associated with user training and quality assurance of these systems in the clinic. With machine learning techniques continuing to evolve and CAD applications expanding to new stages of the patient care process, the current task group report considers the broader issues common to the development of most, if not all, CAD‐AI applications and their translation from the bench to the clinic. The goal is to bring attention to the proper training and validation of machine learning algorithms that may improve their generalizability and reliability and accelerate the adoption of CAD‐AI systems for clinical decision support.

show abstract

WeLineation: crowdsourcing delineations for reliable ground truth estimation

Cited by 6 publications

References 0 publications

A Survey on Deep Learning for Skin Lesion Segmentation

A Survey on Deep Learning for Skin Lesion Segmentation

STAPLE performance assessed on crowdsourced sclera segmentations

AAPM task group report 273: Recommendations on best practices for AI and machine learning for computer‐aided diagnosis in medical imaging

Contact Info

Product

Resources

About