Siamese neural networks for continuous disease severity evaluation and change detection in medical imaging

Abstract

Using medical images to evaluate disease severity and change over time is a routine and important task in clinical decision making. Grading systems are often used, but are unreliable as domain experts disagree on disease severity category thresholds. These discrete categories also do not reflect the underlying continuous spectrum of disease severity. To address these issues, we developed a convolutional Siamese neural network approach to evaluate disease severity at single time points and change between longitudinal patient visits on a continuous spectrum. We demonstrate this in two medical imaging domains: retinopathy of prematurity (ROP) in retinal photographs and osteoarthritis in knee radiographs. Our patient cohorts consist of 4861 images from 870 patients in the Imaging and Informatics in Retinopathy of Prematurity (i-ROP) cohort study and 10,012 images from 3021 patients in the Multicenter Osteoarthritis Study (MOST), both of which feature longitudinal imaging data. Multiple expert clinician raters ranked 100 retinal images and 100 knee radiographs from excluded test sets for severity of ROP and osteoarthritis, respectively. The Siamese neural network output for each image in comparison to a pool of normal reference images correlates with disease severity rank (ρ = 0.87 for ROP and ρ = 0.89 for osteoarthritis), both within and between the clinical grading categories. Thus, this output can represent the continuous spectrum of disease severity at any single time point. The difference in these outputs can be used to show change over time. Alternatively, paired images from the same patient at two time points can be directly compared using the Siamese neural network, resulting in an additional continuous measure of change between images. Importantly, our approach does not require manual localization of the pathology of interest and requires only a binary label for training (same versus different). The location of disease and site of change detected by the algorithm can be visualized using an occlusion sensitivity map-based approach. For a longitudinal binary change detection task, our Siamese neural networks achieve test set receiving operator characteristic area under the curves (AUCs) of up to 0.90 in evaluating ROP or knee osteoarthritis change, depending on the change detection strategy. The overall performance on this binary task is similar compared to a conventional convolutional deep-neural network trained for multi-class classification. Our results demonstrate that convolutional Siamese neural networks can be a powerful tool for evaluating the continuous spectrum of disease severity and change in medical imaging.npj Digital Medicine (2020) 3:48 ; https://doi.

Automated Assessment and Tracking of COVID-19 Pulmonary Disease Severity on Chest Radiographs Using Convolutional Siamese Neural Networks

“…One possible solution to these challenges is to train a convolutional Siamese neural network to estimate radiographic disease severity on a continuous spectrum ( 7 ). Siamese neural networks take two separate images as inputs, which are passed through twinned neural networks ( 8 , 9 ).…”

Differentiation of COVID-19 conditions in planar chest radiographs using optimized convolutional neural networks

“…Occlusion sensitivity maps provide localization information about certain specific areas in the image, when a grayscale mask is placed through the whole CXR image generating a probability map [10,21]. It is used to analyse the network sensitivity to the occlusions of image regions [35]. These maps provide high spatial resolution and finer details.…”

Clinical deployment and validation of a radiology artificial intelligence system for COVID-19

“…We implement a deep learning model that assesses pulmonary consolidative disease severity on chest radiographs of COVID-19 patients 43 . The algorithm incorporates a convolutional Siamese neural network-based approach 44 to provide a measure of disease severity 14 . This approach can help clinicians assess risk for worsening clinical outcomes and can also be used to track change over time from one radiograph to another.…”

“…Monitoring the health of the live inference system is critical. We use the Clara Management Console 13 , the Clara Deploy CLI 14 , and the kubectl 15 tool to monitor all jobs initiated by the DICOM adapter. The console provides a user interface that lists new jobs as they are created, their current status, and upon inference completion, it lists the job duration.…”

“…Real-time reverse transcription-polymerase chain reaction (rRT-PCR) has been widely used for detecting, tracking, and studying COVID-19 across healthcare systems; however, the concerns about availability and stability of these tests 8 along with their moderate sensitivity in diagnosis [9][10][11] has motivated clinicians to seek complimentary tools for diagnosis, prognosis and treatment of this disease. To this end, medical imaging has been used to increase diagnostic accuracy 12,13 , evaluate disease severity 14,15 , and follow up on infected patients 16 .…”