Automatic segmentation of hepatic metastases on DWI images based on a deep learning method: assessment of tumor treatment response according to the RECIST 1.1 criteria

Abstract

Background Evaluation of treated tumors according to Response Evaluation Criteria in Solid Tumors (RECIST) criteria is an important but time-consuming task in medical imaging. Deep learning methods are expected to automate the evaluation process and improve the efficiency of imaging interpretation. Objective To develop an automated algorithm for segmentation of liver metastases based on a deep learning method and assess its efficacy for treatment response assessment according to the RECIST 1.1 criteria. Methods One hundred and sixteen treated patients with clinically confirmed liver metastases were enrolled. All patients had baseline and post-treatment MR images. They were divided into an initial (n = 86) and validation cohort (n = 30) according to the examined time. The metastatic foci on DWI images were annotated by two researchers in consensus. Then the treatment responses were assessed by the two researchers according to RECIST 1.1 criteria. A 3D U-Net algorithm was trained for automated liver metastases segmentation using the initial cohort. Based on the segmentation of liver metastases, the treatment response was assessed automatically with a rule-based program according to the RECIST 1.1 criteria. The segmentation performance was evaluated using the Dice similarity coefficient (DSC), volumetric similarity (VS), and Hausdorff distance (HD). The area under the curve (AUC) and Kappa statistics were used to assess the accuracy and consistency of the treatment response assessment by the deep learning model and compared with two radiologists [attending radiologist (R1) and fellow radiologist (R2)] in the validation cohort. Results In the validation cohort, the mean DSC, VS, and HD were 0.85 ± 0.08, 0.89 ± 0.09, and 25.53 ± 12.11 mm for the liver metastases segmentation. The accuracies of R1, R2 and automated segmentation-based assessment were 0.77, 0.65, and 0.74, respectively, and the AUC values were 0.81, 0.73, and 0.83, respectively. The consistency of treatment response assessment based on automated segmentation and manual annotation was moderate [K value: 0.60 (0.34–0.84)]. Conclusion The deep learning-based liver metastases segmentation was capable of evaluating treatment response according to RECIST 1.1 criteria, with comparable results to the junior radiologist and superior to that of the fellow radiologist.

Kappa statistic considerations in evaluating inter-rater reliability between two raters: which, when and context matters

“…Liu et al [ 3 ] assessed the level of agreement between two raters for a set of ordered categorical variables comprising three categories: PR, SD, and PD. Weighted Kappa is a more appropriate option in this scenario as opposed to Cohen’s Kappa.…”

“…To address this, researchers developed computer-aided systems for automated lesion segmentation. Liu et al [ 3 ] proposed a deep learning-based liver metastases segmentation method that assessed treatment response based on RECIST 1.1 and compared the accuracy of automated segmentation to radiologists’ readings. While the authors’ statement had some merit, the approach requires further evaluation.…”

“… The data has been cited from the article published by Liu et al [ 3 ]. R1: an attending radiologist with 8 year’s reading experience; R2: a fellow radiologist with 4 year’s reading experience; κ c : Cohen’s Kappa value; κ lw : linear weighted Kappa value; κ qw : quadratic weighted Kappa value; κ*: Kappa value calculated by Liu et al; CI: confidence interval; PR: partial response; SD: stable disease; PD: progressive disease …”

“…Unfortunately, some studies misuse statistical procedures, fail to report critical information necessary to interpret their results, or do not adequately address how IRR affects the power of subsequent analyses for hypothesis testing [ 2 ]. This matters arising paper examines the recent publication by Liu et al in BMC Cancer [ 3 ], highlighting controversy of the Kappa statistic and methodological concerns related to IRR assessment. The focus is on the selection of appropriate Kappa statistics, as well as computation, interpretation, and reporting of two commonly-used IRR statistics between two raters.…”