3D/2D model-to-image registration by imitation learning for cardiac procedures

Tóth, Dániel; Miao, Shun; Kurzendorfer, Tanja; Rinaldi, Christopher Aldo; Liao, Rui; Mansi, Tommaso; Rhode, Kawal; Mountney, Peter

doi:10.1007/s11548-018-1774-y

Cited by 39 publications

(30 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Alternatively, motion estimation can be recast as a datadriven learning task (6)(7)(8)(9)(10)(11)(12)(13)(14)(15)(16)(17)(18), which reduces processing times drastically because trained methods can quickly compute centering the images about the center of the left ventricle and cropping the resulting images to the size 80 3 80 3 16. For each subgroup with 30 participants, 20 were randomly chosen for training and the remaining 10 were used for testing, leaving 100 participants for training and 50 participants for testing.…”

mentioning

confidence: 99%

Implementation and Validation of a Three-dimensional Cardiac Motion Estimation Network

Morales¹,

Izquierdo‐Garcia²,

Aganj³

et al. 2019

Radiology: Artificial Intelligence

View full text Add to dashboard Cite

To describe an unsupervised three-dimensional cardiac motion estimation network (CarMEN) for deformable motion estimation from two-dimensional cine MR images. Materials and Methods: A function was implemented using CarMEN, a convolutional neural network that takes two three-dimensional input volumes and outputs a motion field. A smoothness constraint was imposed on the field by regularizing the Frobenius norm of its Jacobian matrix. CarMEN was trained and tested with data from 150 cardiac patients who underwent MRI examinations and was validated on synthetic (n = 100) and pediatric (n = 33) datasets. CarMEN was compared to five state-of-the-art nonrigid body registration methods by using several performance metrics, including Dice similarity coefficient (DSC) and end-point error. Results: On the synthetic dataset, CarMEN achieved a median DSC of 0.85, which was higher than all five methods (minimummaximum median [or MMM], 0.67-0.84; P , .001), and a median end-point error of 1.7, which was lower than (MMM, 2.1-2.7; P , .001) or similar to (MMM, 1.6-1.7; P. .05) all other techniques. On the real datasets, CarMEN achieved a median DSC of 0.73 for Automated Cardiac Diagnosis Challenge data, which was higher than (MMM, 0.33; P , .0001) or similar to (MMM, 0.72-0.75; P. .05) all other methods, and a median DSC of 0.77 for pediatric data, which was higher than (MMM, 0.71-0.76; P , .0001) or similar to (MMM, 0.77-0.78; P. .05) all other methods. All P values were derived from pairwise testing. For all other metrics, CarMEN achieved better accuracy on all datasets than all other techniques except for one, which had the worst motion estimation accuracy. Conclusion: The proposed deep learning-based approach for three-dimensional cardiac motion estimation allowed the derivation of a motion model that balances motion characterization and image registration accuracy and achieved motion estimation accuracy comparable to or better than that of several state-of-the-art image registration algorithms.

show abstract

mentioning

confidence: 99%

Implementation and Validation of a Three-dimensional Cardiac Motion Estimation Network

Morales¹,

Izquierdo‐Garcia²,

Aganj³

et al. 2019

Radiology: Artificial Intelligence

View full text Add to dashboard Cite

show abstract

“…Such methods partially or completely skip the step of precise modeling of image formation or similarity, and instead build up the knowledge in a data-driven manner. We have identified 22 studies that describe methods for direct parameter regression ( Chou and Pizer, 2013 , 2014 ; Chou et al, 2013 ; Zhao et al, 2014 ; Mitrovi et al, 2015 ; Wu et al, 2015 ; Miao et al, 2016a ; Miao et al, 2016b ; Hou et al, 2017 ; Pei et al, 2017 ; Xie et al, 2017 ; Hou et al, 2018 ; Miao et al, 2018 ; Toth et al, 2018 ; Zhang et al, 2018 ; Zheng et al, 2018 ; Guan et al, 2019 , 2020 ; Foote et al, 2019 ; Gao et al, 2020c ; Li et al, 2020 ; Xiangqian et al, 2020 ).…”

Section: Systematic Reviewmentioning

confidence: 99%

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

Unberath

Gao

et al. 2021

Front. Robot. AI

View full text Add to dashboard Cite

Image-based navigation is widely considered the next frontier of minimally invasive surgery. It is believed that image-based navigation will increase the access to reproducible, safe, and high-precision surgery as it may then be performed at acceptable costs and effort. This is because image-based techniques avoid the need of specialized equipment and seamlessly integrate with contemporary workflows. Furthermore, it is expected that image-based navigation techniques will play a major role in enabling mixed reality environments, as well as autonomous and robot-assisted workflows. A critical component of image guidance is 2D/3D registration, a technique to estimate the spatial relationships between 3D structures, e.g., preoperative volumetric imagery or models of surgical instruments, and 2D images thereof, such as intraoperative X-ray fluoroscopy or endoscopy. While image-based 2D/3D registration is a mature technique, its transition from the bench to the bedside has been restrained by well-known challenges, including brittleness with respect to optimization objective, hyperparameter selection, and initialization, difficulties in dealing with inconsistencies or multiple objects, and limited single-view performance. One reason these challenges persist today is that analytical solutions are likely inadequate considering the complexity, variability, and high-dimensionality of generic 2D/3D registration problems. The recent advent of machine learning-based approaches to imaging problems that, rather than specifying the desired functional mapping, approximate it using highly expressive parametric models holds promise for solving some of the notorious challenges in 2D/3D registration. In this manuscript, we review the impact of machine learning on 2D/3D registration to systematically summarize the recent advances made by introduction of this novel technology. Grounded in these insights, we then offer our perspective on the most pressing needs, significant open problems, and possible next steps.

show abstract

“…To this end, the agent would need to be implemented as a parameterized actor. The agent would then estimate the action directly for each given state 25 , 32 , 33 .…”

Section: Introductionmentioning

confidence: 99%

Deep action learning enables robust 3D segmentation of body organs in various CT and MRI images

Zhong

Amrehn

Ravikumar

et al. 2021

Sci Rep

View full text Add to dashboard Cite

In this study, we propose a novel point cloud based 3D registration and segmentation framework using reinforcement learning. An artificial agent, implemented as a distinct actor based on value networks, is trained to predict the optimal piece-wise linear transformation of a point cloud for the joint tasks of registration and segmentation. The actor network estimates a set of plausible actions and the value network aims to select the optimal action for the current observation. Point-wise features that comprise spatial positions (and surface normal vectors in the case of structured meshes), and their corresponding image features, are used to encode the observation and represent the underlying 3D volume. The actor and value networks are applied iteratively to estimate a sequence of transformations that enable accurate delineation of object boundaries. The proposed approach was extensively evaluated in both segmentation and registration tasks using a variety of challenging clinical datasets. Our method has fewer trainable parameters and lower computational complexity compared to the 3D U-Net, and it is independent of the volume resolution. We show that the proposed method is applicable to mono- and multi-modal segmentation tasks, achieving significant improvements over the state-of-the-art for the latter. The flexibility of the proposed framework is further demonstrated for a multi-modal registration application. As we learn to predict actions rather than a target, the proposed method is more robust compared to the 3D U-Net when dealing with previously unseen datasets, acquired using different protocols or modalities. As a result, the proposed method provides a promising multi-purpose segmentation and registration framework, particular in the context of image-guided interventions.

show abstract

3D/2D model-to-image registration by imitation learning for cardiac procedures

Cited by 39 publications

References 16 publications

Implementation and Validation of a Three-dimensional Cardiac Motion Estimation Network

Implementation and Validation of a Three-dimensional Cardiac Motion Estimation Network

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

Deep action learning enables robust 3D segmentation of body organs in various CT and MRI images

Contact Info

Product

Resources

About