Poor agreement between the automated risk assessment of a smartphone application for skin cancer detection and the rating by dermatologists

Abstract

Background Several smartphone applications (app) with an automated risk assessment claim to be able to detect skin cancer at an early stage. Various studies that have evaluated these apps showed mainly poor performance. However, all studies were done in patients and lesions were mainly selected by a specialist. Objectives To investigate the performance of the automated risk assessment of an app by comparing its assessment to that of a dermatologist in lesions selected by the participants. Methods Participants of a National Skin Cancer Day were enrolled in a multicentre study. Skin lesions indicated by the participants were analysed by the automated risk assessment of the app prior to blinded rating by the dermatologist. The ratings of the automated risk assessment were compared to the assessment and diagnosis of the dermatologist. Due to the setting of the Skin Cancer Day, lesions were not verified by histopathology. Results We included 125 participants (199 lesions). The app was not able to analyse 90 cases (45%) of which nine BCC, four atypical naevi and one lentigo maligna. Thirty lesions (67%) with a high and 21 with a medium risk (70%) rating by the app were diagnosed as benign naevi or seborrhoeic keratoses. The interobserver agreement between the ratings of the automated risk assessment and the dermatologist was poor (weighted kappa = 0.02; 95% CI −0.08‐0.12; P = 0.74). Conclusions The rating of the automated risk assessment was poor. Further investigations about the diagnostic accuracy in real‐life situations are needed to provide consumers with reliable information about this healthcare application.

The evolving field of Dermato‐oncology and the role of dermatologists: Position Paper of the EADO, EADV and Task Forces, EDF, IDS, EBDV–UEMS and EORTC Cutaneous Lymphoma Task Force

“…Mobile apps and other electronic resources have the potential to improve access and outcomes for skin cancer patients, but these resources must be utilized with appropriate protocols and guidelines and with caution, as patients need to be educated about their benefits and limitations. Current algorithm‐based smartphone apps to assess images of skin tumours suspicious for skin cancer cannot be relied upon to detect malignant tumours according to the results of published studies 93,94 . Further testing must be conducted in real‐life situations to evaluate the reliability of these healthcare apps for skin cancer used by patients.…”

Interventions to improve early cancer diagnosis of symptomatic individuals: a scoping review

“…42 The ineffective targets or benchmarks were the New Zealand Ministry of Health's 'faster cancer treatment' standards of service provision for melanoma patients, with a target of histopathological diagnosis of melanoma reported within five working days in 80% of cases, and all cases reported in 10 working days. 15 In addition, two unpublished articles from Canada 105 and the UK 107 Ten of these articles were from the UK, 22 29 37 38 51 57 62 65 66 91 two articles were from New Zealand 79 82 and one article each was from Denmark, 89 Netherlands, 21 Italy, 16 India 87 and Spain. 58 These publications focused on varied cancer types in adult patient populations, with two also involving paediatric patients.…”

Use of mobile applications in dermatology

“…[ 17 ] A recent study from the Netherlands involving 199 lesions also concluded that the smartphone based rating of the automated risk assessment for basal cell carcinoma was poor. [ 18 ] A review of two cohorts of lesions published in two studies regarding the use of smartphone applications for triaging suspicious melanoma concluded that accuracy of these applications was low and that there were a significant number of missed melanomas. [ 19 ] A study using simulated patients to evaluate telemedicine websites and apps conclude that there are concerns with respect to transparency, choice, thoroughness, diagnostic, and therapeutic quality.…”