Deep learning‐based motion tracking using ultrasound images

Dai, Xianjin; Lei, Yang; Roper, Justin; Chen, Yue; Bradley, Jeffrey D.; Liu, Tian; Yang, Xiaofeng

doi:10.1002/mp.15321

Cited by 19 publications

(7 citation statements)

References 69 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, to bridge the gap between the phantom trials and real patients, some practical factors should also be properly considered, for example, patient physiological movements during the scans. The research community has already noted these problems, and there are some emerging articles focusing on monitoring and compensating for rigid motion (Jiang et al, 2021b(Jiang et al, , 2022a, articulated motion (Jiang et al, 2022b), and breathing motion (Dai et al, 2021).…”

Section: Discussionmentioning

confidence: 99%

Intelligent robotic sonographer: Mutual information-based disentangled reward learning from few demonstrations

Jiang,

Bi,

Zhou

et al. 2024

The International Journal of Robotics Research

View full text Add to dashboard Cite

Ultrasound (US) imaging is widely used for biometric measurement and diagnosis of internal organs due to the advantages of being real-time and radiation-free. However, due to inter-operator variations, resulting images highly depend on the experience of sonographers. This work proposes an intelligent robotic sonographer to autonomously “explore” target anatomies and navigate a US probe to standard planes by learning from the expert. The underlying high-level physiological knowledge from experts is inferred by a neural reward function, using a ranked pairwise image comparison approach in a self-supervised fashion. This process can be referred to as understanding the “language of sonography.” Considering the generalization capability to overcome inter-patient variations, mutual information is estimated by a network to explicitly disentangle the task-related and domain features in latent space. The robotic localization is carried out in coarse-to-fine mode based on the predicted reward associated with B-mode images. To validate the effectiveness of the proposed reward inference network, representative experiments were performed on vascular phantoms (“line” target), two types of ex vivo animal organ phantoms (chicken heart and lamb kidney representing “point” target), and in vivo human carotids. To further validate the performance of the autonomous acquisition framework, physical robotic acquisitions were performed on three phantoms (vascular, chicken heart, and lamb kidney). The results demonstrated that the proposed advanced framework can robustly work on a variety of seen and unseen phantoms as well as in vivo human carotid data. Code: https://github.com/yuan-12138/MI-GPSR . Video: https://youtu.be/u4ThAA9onE0 .

show abstract

Section: Discussionmentioning

confidence: 99%

Intelligent robotic sonographer: Mutual information-based disentangled reward learning from few demonstrations

Jiang,

Bi,

Zhou

et al. 2024

The International Journal of Robotics Research

View full text Add to dashboard Cite

show abstract

“…To have more evaluations, we conducted the extra experiments on the public dataset of cardiac acquisitions for multistructure ultrasound segmentation (CAMUS) that was used in our previous study [31]. The CAMUS dataset includes 2D US images from 450 patients and meanwhile contains expert annotations in the left atrium.…”

Section: Discussionmentioning

confidence: 99%

“…However, these methods are often lost in the similar image structures out of interest regions or missing the temporal features between frames. Dai et al [31] developed a Markov-like network, which is implemented via generative adversarial networks, to extract features from sequential US frames and thereby estimate a set of deformation vector fields (DVFs) through the registration of the tracked frame and the untracked frames. Finally, they determined the positions of the landmarks in the untracked frames by shifting landmarks in the tracked frame according to the estimated DVFs [32].…”

Section: Introductionmentioning

confidence: 99%

Landmark tracking in liver US images using cascade convolutional neural networks with long short-term memory

Zhang

Dai

Tian

et al. 2023

Meas. Sci. Technol.

Self Cite

View full text Add to dashboard Cite

This study proposed a deep learning-based tracking method for ultrasound-guided radiation therapy. The proposed cascade deep learning model is composed of an attention network, a mask region-based convolutional neural network (mask R-CNN), and a long short-term memory (LSTM) network. The attention network learns a mapping from a US image to a suspected area of landmark motion in order to reduce the search region. The mask R-CNN then produces multiple region-of-interest (ROI) proposals in the reduced region and identifies the proposed landmark via three network heads: bounding box regression, proposal classification, and landmark segmentation. The LSTM network models the temporal relationship among the successive image frames for bounding box regression and proposal classification. To consolidate the final proposal, a selection method is designed according to the similarities between sequential frames. The proposed method was tested on the liver US tracking datasets used in the Medical Image Computing and Computer Assisted Interventions (MICCAI) 2015 challenges, where the landmarks were annotated by three experienced observers to obtain their mean positions. Five-fold cross-validation on the 24 given US sequences with ground truths shows that the mean tracking error for all landmarks is 0.65 ± 0.56 mm, and the errors of all landmarks are within 2 mm. We further tested the proposed model on 69 landmarks from the testing dataset that has a similar image pattern to the training pattern, resulting in a mean tracking error of 0.94 ± 0.83 mm. The proposed deep-learning model was implemented on a GPU, tracking 47 to 81 frames per second. Our experimental results have demonstrated the feasibility and accuracy of our proposed method in tracking liver anatomic landmarks using US images, providing a potential solution for real-time liver tracking for active motion management during radiation therapy.

show abstract

“…[ 13 ] incorporated on-line learning of a supporter model that captured the coupling of motion between image features, making it potentially useful for predicting target positions, which can be individually tracked. Further works, including [ 14 , 15 , 16 ], aimed to more explicitly incorporate temporal motion information through Conv-LSTMs, PCA motion models and a GAN-based Markov-like net that incorporates transformer modules respectively.…”

Section: Introductionmentioning

confidence: 99%

Robust and Realtime Large Deformation Ultrasound Registration Using End-to-End Differentiable Displacement Optimisation

Heinrich

Siebert

Graf

et al. 2023

Sensors

View full text Add to dashboard Cite

Image registration for temporal ultrasound sequences can be very beneficial for image-guided diagnostics and interventions. Cooperative human–machine systems that enable seamless assistance for both inexperienced and expert users during ultrasound examinations rely on robust, realtime motion estimation. Yet rapid and irregular motion patterns, varying image contrast and domain shifts in imaging devices pose a severe challenge to conventional realtime registration approaches. While learning-based registration networks have the promise of abstracting relevant features and delivering very fast inference times, they come at the potential risk of limited generalisation and robustness for unseen data; in particular, when trained with limited supervision. In this work, we demonstrate that these issues can be overcome by using end-to-end differentiable displacement optimisation. Our method involves a trainable feature backbone, a correlation layer that evaluates a large range of displacement options simultaneously and a differentiable regularisation module that ensures smooth and plausible deformation. In extensive experiments on public and private ultrasound datasets with very sparse ground truth annotation the method showed better generalisation abilities and overall accuracy than a VoxelMorph network with the same feature backbone, while being two times faster at inference.

show abstract

Deep learning‐based motion tracking using ultrasound images

Cited by 19 publications

References 69 publications

Intelligent robotic sonographer: Mutual information-based disentangled reward learning from few demonstrations

Intelligent robotic sonographer: Mutual information-based disentangled reward learning from few demonstrations

Landmark tracking in liver US images using cascade convolutional neural networks with long short-term memory

Robust and Realtime Large Deformation Ultrasound Registration Using End-to-End Differentiable Displacement Optimisation

Contact Info

Product

Resources

About