Evaluation of current algorithms for segmentation of scar tissue from late Gadolinium enhancement cardiovascular magnetic resonance of the left atrium: an open-access grand challenge

Abstract

BackgroundLate Gadolinium enhancement (LGE) cardiovascular magnetic resonance (CMR) imaging can be used to visualise regions of fibrosis and scarring in the left atrium (LA) myocardium. This can be important for treatment stratification of patients with atrial fibrillation (AF) and for assessment of treatment after radio frequency catheter ablation (RFCA). In this paper we present a standardised evaluation benchmarking framework for algorithms segmenting fibrosis and scar from LGE CMR images. The algorithms reported are the response to an open challenge that was put to the medical imaging community through an ISBI (IEEE International Symposium on Biomedical Imaging) workshop.MethodsThe image database consisted of 60 multicenter, multivendor LGE CMR image datasets from patients with AF, with 30 images taken before and 30 after RFCA for the treatment of AF. A reference standard for scar and fibrosis was established by merging manual segmentations from three observers. Furthermore, scar was also quantified using 2, 3 and 4 standard deviations (SD) and full-width-at-half-maximum (FWHM) methods. Seven institutions responded to the challenge: Imperial College (IC), Mevis Fraunhofer (MV), Sunnybrook Health Sciences (SY), Harvard/Boston University (HB), Yale School of Medicine (YL), King’s College London (KCL) and Utah CARMA (UTA, UTB). There were 8 different algorithms evaluated in this study.ResultsSome algorithms were able to perform significantly better than SD and FWHM methods in both pre- and post-ablation imaging. Segmentation in pre-ablation images was challenging and good correlation with the reference standard was found in post-ablation images. Overlap scores (out of 100) with the reference standard were as follows: Pre: IC = 37, MV = 22, SY = 17, YL = 48, KCL = 30, UTA = 42, UTB = 45; Post: IC = 76, MV = 85, SY = 73, HB = 76, YL = 84, KCL = 78, UTA = 78, UTB = 72.ConclusionsThe study concludes that currently no algorithm is deemed clearly better than others. There is scope for further algorithmic developments in LA fibrosis and scar quantification from LGE CMR images. Benchmarking of future scar segmentation algorithms is thus important. The proposed benchmarking framework is made available as open-source and new participants can evaluate their algorithms via a web-based interface.

A fully automatic deep learning method for atrial scarring segmentation from late gadolinium-enhanced MRI images

“…Compared to existing methods with manually delineated LA+PVs, our method obtained significantly higher Dice scores (Figure 4). The four methods we compared in this study were described in the benchmarking work [9]; however, we only implemented standard versions of these algorithms and did not perform further optimization. Moreover, the datasets for which those algorithms were tuned are different from those used in our study; therefore, a totally fair comparison may not be possible although similar performance can be observed between our implementation and those reported in [9] especially for the pre-ablation cases.…”

“…The four methods we compared in this study were described in the benchmarking work [9]; however, we only implemented standard versions of these algorithms and did not perform further optimization. Moreover, the datasets for which those algorithms were tuned are different from those used in our study; therefore, a totally fair comparison may not be possible although similar performance can be observed between our implementation and those reported in [9] especially for the pre-ablation cases. For post-ablation cases, our SAS results showed similar results to the best performed method reported in [9] but with smaller variance, which may be due to the fact that the datasets used in the benchmarking work were acquired from multiple institutions.…”

“…We then applied a morphological dilation (assuming LA wall thickness is 3mm) to extract the LA wall and PVs. The blood pool regions were extracted by a morphological erosion (5mm) from the endocardial LA boundary and the pixel intensities were normalized according to the blood pool intensities [9].…”

“…Perry et al [8] applied k-means clustering to quantitatively assess normal and scarred tissue from manual defined LA and PVs geometry. Karim et al [12] k-means clustering, and graph-cuts methods to establish a benchmarking work [9]. The benchmarking work was tested on data acquired from multiple institutions, and the LA endocardium and cavity for each scan was also provided to all the participants.…”

Closing the Knowledge Gaps

“…Joint and collaborative efforts toward standardization and validation of LA imaging protocols are needed and underway. A recent study by Karim et al 16 offered a challenge to compete for the best magnetic resonance imaging LA fibrosis algorithm with participation of 7 leading centers in the field assessing 8 imaging algorithms. Sixty magnetic resonance imaging scans were used to compare the performance of each center's scar segmentation algorithm(s).…”