We have attempted to reproduce the results in Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs , published in JAMA 2016; 316(22), using publicly available data sets. We re-implemented the main method in the original study since the source code is not available. The original study used non-public fundus images from EyePACS and three hospitals in India for training. We used a different EyePACS data set from Kaggle. The original study used the benchmark data set Messidor-2 to evaluate the algorithm’s performance. We used another distribution of the Messidor-2 data set, since the original data set is no longer available. In the original study, ophthalmologists re-graded all images for diabetic retinopathy, macular edema, and image gradability. We have one diabetic retinopathy grade per image for our data sets, and we assessed image gradability ourselves. We were not able to reproduce the original study’s results with publicly available data. Our algorithm’s area under the receiver operating characteristic curve (AUC) of 0.951 (95% CI, 0.947-0.956) on the Kaggle EyePACS test set and 0.853 (95% CI, 0.835-0.871) on Messidor-2 did not come close to the reported AUC of 0.99 on both test sets in the original study. This may be caused by the use of a single grade per image, or different data. This study shows the challenges of reproducing deep learning method results, and the need for more replication and reproduction studies to validate deep learning methods, especially for medical image analysis. Our source code and instructions are available at: https://github.com/mikevoets/jama16-retina-replication .

No abstract

Skin cancer is one of the most common types of cancer. Skin cancers are classified as nonmelanoma and melanoma, with the first type being the most frequent and the second type being the most deadly. The key to effective treatment of skin cancer is early detection. With the recent increase of computational power, the number of algorithms to detect and classify skin lesions has increased. The overall verdict on systems based on clinical and dermoscopic images captured with conventional RGB (red, green, and blue) cameras is that they do not outperform dermatologists.Computer-based systems based on conventional RGB images seem to have reached an upper limit in their performance, while emerging technologies such as hyperspectral and multispectral imaging might possibly improve the results. These types of images can explore spectral regions beyond the human eye capabilities. Feature selection and dimensionality reduction are crucial parts of extracting salient information from this type of data. It is necessary to extend current classification methodologies to use all of the spatiospectral information, and deep learning models should be explored since they are capable of learning robust feature detectors from data. There is a lack of large, high-quality datasets of hyperspectral skin lesion images, and there is a need for tools that can aid with monitoring the evolution of skin lesions over time. To understand the rich information contained in hyperspectral images, further research using data science and statistical methodologies, such as functional data analysis, scale-space theory, machine learning, and so on, are essential. This article is categorized under: Applications of Computational Statistics > Health and Medical Data/Informatics K E Y W O R D S hyperspectral, machine learning, melanoma, skin cancer

The present segmentation algorithm is fast and intuitive. It gives correct segmentation for most types of skin lesions, but fails when the lesion is brighter than the surrounding skin.

Commercially available clinical decision support systems (CDSSs) for skin cancer have been designed for the detection of melanoma only. Correct use of the systems requires expert knowledge, hampering their utility for nonexperts. Furthermore, there are no systems to detect other common skin cancer types, that is, nonmelanoma skin cancer (NMSC). As early diagnosis of skin cancer is essential, there is a need for a CDSS that is applicable to all types of skin lesions and is suitable for nonexperts. Nevus Doctor (ND) is a CDSS being developed by the authors. We here investigate ND's ability to detect both melanoma and NMSC and the opportunities for improvement. An independent test set of dermoscopic images of 870 skin lesions, including 44 melanomas and 101 NMSCs, were analysed by ND. Its sensitivity to melanoma and NMSC was compared to that of Mole Expert (ME), a commercially available CDSS, using the same set of lesions. ND and ME had similar sensitivity to melanoma. For ND at 95% melanoma sensitivity, the NMSC sensitivity was 100%, and the specificity was 12%. The melanomas misclassified by ND at 95% sensitivity were correctly classified by ME, and vice versa. ND is able to detect NMSC without sacrificing melanoma sensitivity.