Detecting eczema areas in digital images: an impossible task?

Abstract

Assessing the severity of atopic dermatitis (AD, or eczema) traditionally relies on a face-to-face assessment by healthcare professionals, and may suffer from inter- and intra-rater variability. With the expanding role of telemedicine, several machine learning algorithms have been proposed to automatically assess AD severity from digital images. Those algorithms usually detect and then delineate (“segment”) AD lesions before assessing lesional severity, and are trained using the data of AD areas detected by healthcare professionals. To evaluate the reliability of such data, we estimated the inter-rater reliability of AD segmentation in digital images.Four dermatologists independently segmented AD lesions in 80 digital images collected in a published clinical trial. We estimated the inter-rater reliability of the AD segmentation using the intra-class correlation coefficients (ICCs) at the pixel-level and the area-levels for different resolutions of the images. The average ICC was 0.45 (SE = 0.04) corresponding to a “poor” agreement between raters, while the degree of agreement for AD segmentation varied from image to image.The AD segmentation in digital images is highly rater-dependent even between dermatologists. Such limitations need to be taken into consideration when the AD segmentation data are used to train machine learning algorithms that assess eczema severity.

Development of High-Quality Artificial Intelligence in Dermatology: Guidelines, Pitfalls, and Potential

“…If the AI algorithm is trained using a component of inaccurate eczema severity data, the outcomes will be erroneous. Hurault et al (2022), also recently published in JID Innovations, examined whether high quality eczema segmentation data can be obtained from dermatologists using images reliably. They found inter-rater reliability of eczema segmentation varied from image to image with a poor agreement between the raters on average.…”

“…These results reinforce the difficulty of reliably and consistently detecting atopic dermatitis from photos. Hurault et al (2022) offered suggestions for improving poor inter-rater reliability in segmentation data for machine learning models, including letting the algorithm identify eczema regions by itself, using algorithms that can be trained on noisy segmentation labels, improving the training of the raters, and averaging the segmentation from multiple raters. Alternatively, photo assessments that are performed in person may assist in reliability, given this is the setting in which the tools were validated, including in patients with highly pigmented skin.…”

“…The ground truth labels for lesion segmentation and visual sign classification were obtained by averaging the results of the three annotators. As mentioned by Hurault et al (2022), this is one method that may reduce inter-rater reliability. Although the results by Medula et al (2022) show potential, with lesion segmentation annotation generally consistent across data sets, some visual signs such as edema and dryness were difficult to assess in photos, particularly in patients with darker skin tones.…”

“…Rigorous clinical trials are much needed to evaluate AI algorithms to understand whether they translate to the clinical settings in which they will be used ( Daneshjou, 2022a ; Han et al., 2022 ). In addition to evaluating algorithms that aid in diagnostic support, researchers are trying to study whether AI is able to delineate body surface area and localize lesions of several skin diseases, including vitiligo, psoriasis, and atopic dermatitis ( Guo et al., 2022b ; Han et al., 2022 ; Hurault et al., 2022 ; Medela et al., 2022 ; Schaap et al., 2022 ).…”

“…If the AI algorithm is trained using a component of inaccurate eczema severity data, the outcomes will be erroneous. Hurault et al. (2022) , also recently published in JID Innovations , examined whether high-quality eczema segmentation data can be obtained from dermatologists using images reliably.…”