Deep-learning cardiac motion analysis for human survival prediction

Bello, Ghalib; Dawes, Timothy J W; Duan, Jinming; Biffi, Carlo; Marvao, Antonio de; Howard, Luke; Gibbs, J. Simon R.; Wilkins, Martin R.; Cook, Stuart A.; Rueckert, Daniel; O’Regan, Declan P

doi:10.1038/s42256-019-0019-2

Cited by 197 publications

(143 citation statements)

References 71 publications

Supporting

Mentioning

142

Contrasting

Unclassified

Order By: Relevance

“…Previous studies of deep learning on medical imaging focused on resource-intensive imaging modalities common in resource-rich settings 37,38 or sub-speciality imaging with focused indication. 12,13,16 These modalities often need retrospective annotation by experts as the clinical workflow often does not require detailed measurements or localizations.…”

Section: Discussionmentioning

confidence: 99%

Deep Learning Interpretation of Echocardiograms

Ghorbani

Ouyang

Abid

et al. 2019

Preprint

View full text Add to dashboard Cite

Echocardiography uses ultrasound technology to capture high temporal and spatial resolution images of the heart and surrounding structures and is the most common imaging modality in cardiovascular medicine. Using convolutional neural networks on a large new dataset, we show that deep learning applied to echocardiography can identify local cardiac structures, estimate cardiac function, and predict systemic phenotypes that modify cardiovascular risk but not readily identifiable to human interpretation. Our deep learning model, EchoNet, accurately identified the presence of pacemaker leads (AUC = 0.89), enlarged left atrium (AUC = 0.85), normal left ventricular wall thickness (AUC = 0.75), left ventricular end systolic and diastolic volumes(R 2 = 0.73 and R 2 = 0.68), and ejection fraction (R 2 = 0.48) as well as predicted systemic phenotypes of age (R 2 = 0.46), sex (AUC = 0.88), weight (R 2 = 0.56), and height (R 2 = 0.33). Interpretation analysis validates that EchoNet shows appropriate attention to key cardiac structures when performing human-explainable tasks and highlight hypothesis-generating regions of interest when predicting systemic phenotypes difficult for human interpretation. Machine learning on echocardiography images can streamline repetitive tasks in the clinical workflow, standardize interpretation in areas with insufficient qualified cardiologists, and more consistently produce echocardiographic measurements.Related works Current literature have already shown that it is possible to identify standard echocardiogram views from unlabeled datasets. 5,6,24 Previous works have used convolutional neural networks (CNNs) trained on images and videos from echocardiography to perform segmentation to identify cardiac structures and derive cardiac function. In this study, we extend previous analyses to show that EchoNet, our deep learning model using echocardiography images, local cardiac structures and anatomy, estimate volumetric measurements and metrics of cardiac function, and predict systemic human phenotypes that modify cardiovascular risk. Additionally, we show the first application of interpretation frameworks to understand deep learning models from echocardiogram images. Human identifiable features, such as the presence of pacemaker and defibrillator leads, left ventricular hypertrophy, and abnormal left atrial chamber size identified by our convolutional neural network were validated using interpretation frameworks to highlight the most relevant regions of interest. To the best of our knowledge, we develop the first deep learning model that can directly predict age, sex, weight and height from echocardiogram images and use interpretation methods to understand how the model predicts these systemic phenotypes difficult for human interpreters. 2/14Lessons from model training and experiments EchoNet performance greatly improved with efforts to augment data size, homogenize input data, and with optimize model training with hyperparameter search. Our experience shows that increasing number of unique patien...

show abstract

Section: Discussionmentioning

confidence: 99%

Deep Learning Interpretation of Echocardiograms

Ghorbani

Ouyang

Abid

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…This has two main advantages: 1) template shapes for each disease class can be obtained by sampling from the learned distributions in a top-down fashion (starting from the highest level in the hierarchy p(z L |y) and subsequently from every prior p θ (z i |z i+1 )). The posterior p(z L |y) can be estimated by kernel density estimation and, since z L is typically very lowdimensional, this estimation is straightforward; 2) if the latent space z L is designed to be 2D or 3D, the distributions p(z L |y) in the classification space can be directly visualised without the need of further offline dimensionality reduction techniques required in previous works [32], [33]. […”

Section: Lvae For Interpretable Shape Analysismentioning

confidence: 99%

Explainable Anatomical Shape Analysis Through Deep Hierarchical Generative Models

Biffi

Cerrolaza

Tarroni

et al. 2020

IEEE Trans. Med. Imaging

Self Cite

View full text Add to dashboard Cite

Quantification of anatomical shape changes still relies on scalar global indexes which are largely insensitive to regional or asymmetric modifications. Accurate assessment of pathology-driven anatomical remodeling is a crucial step for the diagnosis and treatment of heart conditions. Deep learning approaches have recently achieved wide success in the analysis of medical images, but they lack interpretability in the feature extraction and decision processes. In this work, we propose a new interpretable deep learning model for shape analysis. In particular, we exploit deep generative networks to model a population of anatomical segmentations through a hierarchy of conditional latent variables. At the highest level of this hierarchy, a two-dimensional latent space is simultaneously optimised to discriminate distinct clinical conditions, enabling the direct visualisation of the classification space. Moreover, the anatomical variability encoded by this discriminative latent space can be visualised in the segmentation space thanks to the generative properties of the model, making the classification task transparent. This approach yielded high accuracy in the categorisation of healthy and remodelled hearts when tested on unseen segmentations from our own multi-centre dataset as well as in an external validation set. More importantly, it enabled the visualisation in three-dimensions of the most discriminative anatomical features between the two conditions. The proposed approach scales effectively to large populations, facilitating highthroughput analysis of normal anatomy and pathology in largescale studies of volumetric imaging.

show abstract

“…Such deep neural networks cannot only be used for disease classification, but also for detecting anatomical structures [29], identifying the correct scan plane in fetal ultrasound images [13], detecting image artefacts in cardiac MR images [83] or assessing the image quality in fetal ultrasound images [118]. Finally, similar approaches can also be used for clinical decision support [35,2,33] or for prediction of patient survival [15].…”

Section: Image Interpretationmentioning

confidence: 99%

Model-Based and Data-Driven Strategies in Medical Image Computing

Rueckert

Schnabel

2020

Proc. IEEE

Self Cite

View full text Add to dashboard Cite

Model-based approaches for image reconstruction, analysis and interpretation have made significant progress over the last decades. Many of these approaches are based on either mathematical, physical or biological models. A challenge for these approaches is the modelling of the underlying processes (e.g. the physics of image acquisition or the patho-physiology of a disease) with appropriate levels of detail and realism. With the availability of large amounts of imaging data and machine learning (in particular deep learning) techniques, data-driven approaches have become more widespread for use in different tasks in reconstruction, analysis and interpretation. These approaches learn statistical models directly from labelled or unlabeled image data and have been shown to be very powerful for extracting clinically useful information from medical imaging. While these data-driven approaches often outperform traditional model-based approaches, their clinical deployment often poses challenges in terms of robustness, generalization ability and interpretability. In this article, we discuss what developments have motivated the shift from model-based approaches towards data-driven strategies and what potential problems are associated with the move towards purely data-driven approaches, in particular deep learning. We also discuss some of the open challenges for data-driven approaches, e.g. generalization to new unseen data (e.g. transfer learning), robustness to adversarial attacks and interpretability. Finally, we conclude with a discussion on how these approaches may lead to the development of more closely coupled imaging pipelines that are optimized in an end-to-end fashion.All steps of the medical imaging pipeline typically make extensive use of models. In this paper we define the term model in a very general fashion, in the sense that it provides a transformation of input (data) into the desired output. For example, in image reconstruction the model helps to transform the acquired sensor data into an image, in image registration the model is used to relate two images (inputs) via a transformation (output), and in image segmentation the model is used to transform an image in which each pixel corresponds to an intensity value, into an semantic segmentation where each pixel has a label corresponding to an organ, anatomical structure or pathology. In diagnosis, the model helps to map image derived features and biomarkers

show abstract

Deep-learning cardiac motion analysis for human survival prediction

Cited by 197 publications

References 71 publications

Deep Learning Interpretation of Echocardiograms

Deep Learning Interpretation of Echocardiograms

Explainable Anatomical Shape Analysis Through Deep Hierarchical Generative Models

Model-Based and Data-Driven Strategies in Medical Image Computing

Contact Info

Product

Resources

About