A Multi-Rate State Observer for Visual Tracking of Magnetic Micro-Agents Using 2D Slow Medical Imaging Modalities

Abstract

Minimally invasive surgery can benefit greatly from utilizing micro-agents. These miniaturized agents need to be clearly visualized and precisely controlled to ensure the success of the surgery. Since medical imaging modalities suffer from low acquisition rate, multi-rate sampling methods can be used to estimate the intersample states of micro-agents. Hence, the sampling rate of the controller can be virtually increased even if the position data is acquired using a slow medical imaging modality. This study presents multi-rate Luenberger and Kalman state estimators for visual tracking of microagents. The micro-agents are tracked using sum of squared differences and normalized cross correlation based visual tracking. Further, the outputs of the two methods are merged to minimize the tracking error and prevent tracking failures. During the experiments, the micro-agents with different geometrical shapes and sizes are imaged using a 2D ultrasound machine and a microscope, and manipulated using electromagnetic coils. The multi-rate state estimation accuracy is measured using a high speed camera. The precision of the tracking and multi-rate state estimation are verified experimentally under challenging conditions. For this purpose, an elliptical shaped magnetic micro-agent with a length of 48 pixels is used. Maximum absolute error in x and y axes are 2.273 and 2.432 pixels for an 8-fold increase of the sample rate (25 frames per second), respectively. During the experiments, it was observed that the micro-agents could be tracked more reliably using normalized cross correlation based visual tracking and intersample states could be estimated more accurately using Kalman state estimator. Experimental results show that the proposed method could be used to track micro-agents in medical imaging modalities and estimate system states at intermediate time instants in real-time.

Visualization of micro-agents and surroundings by real-time multicolor fluorescence microscopy

“…Path information obtained by visual tracking shows the translational motions of the spheroids up to 218 µm/s ( ≈ 1.3 body-length/sec) in Fig. 4c2,d2,e2 41 . In order to visualize the rotational mobility, a spheroid is moved clockwise by a single micro-agent attachment with a maximum magnetic field of 36 mT (Fig.…”

A Flexible Catheter System for Ultrasound-Guided Magnetic Projectile Delivery

“…A stack of position-ordered two-dimensional ultrasound frames is collected during a single sweep. The silhouette of the catheter, spanning between the frames at its base ({B}) and tip ({C}), is extracted from the stack iteratively by a template-based tracker similar to [20]. The PRB model is then reconstructed from the resulting point-cloud by using the technique presented in our previous work [6].…”

Dynamic Visual Tracking for Robot Manipulator Using Adaptive Fading Kalman Filter

“…In the study of visual state estimation, the basic problem is to consider the effect of sensor measurement noise on the extraction of visual features. In [17], for visual tracking of micro-agents in minimally invasive surgery, Kalman state estimators are used to estimate the inter-sample states of micro-agents due to the low acquisition rate of medical imaging modalities. In [18], the extended Kalman filter (EKF) is used to estimate the object velocity state with visual measurement noise, which is applied to grasp a tumbling satellite.…”