P63 Effectiveness of an image-analysing artificial intelligence-based digital health technology to diagnose nonmelanoma skin cancer and benign skin lesions

Abstract

Squamous cell carcinoma (SCC) and basal cell carcinoma (BCC) are common types of nonmelanoma skin cancer (NMSC). The DERM-003 study was a prospective, multicentre, single-arm, masked study that aimed to demonstrate the effectiveness of an artificial intelligence-based digital health technology (AI-DHT) to identify SCC, BCC and premalignant conditions in dermoscopic images of suspicious skin lesions. Patients with at least one suspicious skin lesion that was suitable for photography were eligible. Each lesion was photographed with three smartphone cameras (iPhone 6S, iPhone 11 and Samsung 10) with a dermoscopic lens attached. Each image was assessed by the AI-DHT. A clinical diagnosis was made by the dermatologist, and histopathology results were obtained for biopsied lesions. The AI-DHT output was compared with the histopathology diagnosis, and the area under the receiver operating characteristic curve (AUROC) was calculated. Secondary endpoints included other diagnostic measures and assessment of premalignant lesions. Altogether, 572 patients (49.5% female, mean age 68.5 years, 96.9% Fitzpatrick skin types I–III) were recruited from four UK National Health Service trusts, providing images of 611 suspicious lesions. There were no exclusion criteria relating to skin type. In total, 592 lesions had images from all three cameras available; only one lesion had no images available. Altogether, 395 lesions had a histopathology result. Forty-seven biopsied lesions were diagnosed as SCC and 184 as BCC. The AUROCs for images taken by the iPhone 6S were 0.88 [95% confidence interval (CI) 0.83–0.93] for SCC and 0.87 (95% CI 0.84–0.91) for BCC. For Samsung 10, the AUROCs were 0.85 (95% CI 0.79–0.90) and 0.87 (95% CI 0.83–0.90), and for iPhone 11, they were 0.88 (95% CI 0.84–0.93) and 0.89 (95% CI 0.86–0.92) for SCC and BCC, respectively. Using images taken on iPhone 6S of biopsied-only lesions, the sensitivity rates in detecting SCC and BCC were 98% (95% CI 88–100) and 94% (95% CI 90–97), respectively; the specificity rates were 38% (95% CI 33–44) and 28% (95% CI 21–35), respectively; the positive predictive values were 17% (95% CI 16–19) and 59% (95% CI 56–61), respectively; and the NPVs were 99% (95% CI 95–100) and 81% (95% CI 70–89), respectively. All 12 lesions diagnosed as Bowen disease were classified correctly by the AI-DHT. Of the 61 lesions diagnosed as actinic keratosis, 52 were correctly classified of the 22 lesions diagnosed as dysplastic and 18 were correctly classified by the AI-DHT. All 16 lesions diagnosed as melanoma were classified as such by the AI-DHT. The AI-DHT has the potential to support the diagnosis of NMSC and premalignant lesions.

Real-world post-deployment performance of a novel machine learning-based digital health technology for skin lesion assessment and suggestions for post-market surveillance

“…During this period, no patients were discharged from the service and re-presented later with the same lesion being diagnosed as skin cancer. While other published evidence demonstrate a gap in model performance when evaluating realworld prospective clinical use compared with in silico data (25-27), our analysis demonstrates that DERM can be deployed safely in live clinical services accessible to patients from a broad range of age groups and skin types, with sensitivity and specificity in-line with target thresholds and performance demonstrated in pre-marketing authorisation studies (8)(9)(10)(11)(12)(13).…”

“…Artificial intelligence as a medical device (AIaMD) has the potential to help increase workflow efficiency through triage and supporting clinical decisions in skin cancer pathways (8)(9)(10)(11)(12)(13); however, evidence for AIaMDs has largely reflected performance using retrospective data (13)(14)(15)(16). There remains the need to understand how appropriately regulated AIaMD platforms perform in real-world clinical settings, including how algorithmic improvements or optimisation for different patient populations affects performance over time.…”

“…Deep Ensemble for Recognition of Malignancy (DERM; Skin Analytics, London, UK) is an AIaMD that uses deep learning techniques to assess dermoscopic images of skin lesions, identify features associated with malignancies and support referral decisions for patients ≥18 years (8)(9)(10)(11)(12)(13). DERM is intended to be used for the screening, triage, and assessment of skin lesions, and outputs a suggested diagnosis and referral recommendation.…”