2D/3D image registration to align a 3D volume and 2D X-ray images is a challenging problem due to its ill-posed nature and various artifacts presented in 2D X-ray images. In this paper, we propose a multi-agent system with an auto attention mechanism for robust and efficient 2D/3D image registration. Specifically, an individual agent is trained with dilated Fully Convolutional Network (FCN) to perform registration in a Markov Decision Process (MDP) by observing a local region, and the final action is then taken based on the proposals from multiple agents and weighted by their corresponding confidence levels. The contributions of this paper are threefold. First, we formulate 2D/3D registration as a MDP with observations, actions, and rewards properly defined with respect to X-ray imaging systems. Second, to handle various artifacts in 2D X-ray images, multiple local agents are employed efficiently via FCN-based structures, and an auto attention mechanism is proposed to favor the proposals from regions with more reliable visual cues. Third, a dilated FCN-based training mechanism is proposed to significantly reduce the Degree of Freedom in the simulation of registration environment, and drastically improve training efficiency by an order of magnitude compared to standard CNN-based training method. We demonstrate that the proposed method achieves high robustness on both spine cone beam Computed Tomography data with a low signal-to-noise ratio and data from minimally invasive spine surgery where severe image artifacts and occlusions are presented due to metal screws and guide wires, outperforming other state-of-the-art methods (single agent-based and optimization-based) by a large margin.

No abstract

Computer vision has a wide range of applications from medical image analysis to robotics. Over the past few years, the field has been transformed by machine learning and stands to benefit from potential advances in quantum computing. The main challenge for processing images on current and near-term quantum devices is the size of the data such devices can process. Images can be large, multidimensional and have multiple color channels. Current machine learning approaches to computer vision that exploit quantum resources require a significant amount of manual pre-processing of the images in order to be able to fit them onto the device. This paper proposes a framework to address the problem of processing large scale data on small quantum devices. This framework does not require any dataset-specific processing or information and works on large, grayscale and RGB images. Furthermore, it is capable of scaling to larger quantum hardware architectures as they become available. In the proposed approach, a classical autoencoder is trained to compress the image data to a size that can be loaded onto a quantum device. Then, a Restricted Boltzmann Machine (RBM) is trained on the D-Wave device using the compressed data, and the weights from the RBM are then used to initialize a neural network for image classification. Results are demonstrated on two MNIST datasets and two medical imaging datasets.

2D/3D image registration to align a 3D volume and 2D X-ray images is a challenging problem due to its ill-posed nature and various artifacts presented in 2D X-ray images. In this paper, we propose a multi-agent system with an auto attention mechanism for robust and efficient 2D/3D image registration. Specifically, an individual agent is trained with dilated Fully Convolutional Network (FCN) to perform registration in a Markov Decision Process (MDP) by observing a local region, and the final action is then taken based on the proposals from multiple agents and weighted by their corresponding confidence levels. The contributions of this paper are threefold. First, we formulate 2D/3D registration as a MDP with observations, actions, and rewards properly defined with respect to X-ray imaging systems. Second, to handle various artifacts in 2D X-ray images, multiple local agents are employed efficiently via FCN-based structures, and an auto attention mechanism is proposed to favor the proposals from regions with more reliable visual cues. Third, a dilated FCN-based training mechanism is proposed to significantly reduce the Degree of Freedom in the simulation of registration environment, and drastically improve training efficiency by an order of magnitude compared to standard CNN-based training method. We demonstrate that the proposed method achieves high robustness on both spine cone beam Computed Tomography data with a low signal-to-noise ratio and data from minimally invasive spine surgery where severe image artifacts and occlusions are presented due to metal screws and guide wires, outperforming other state-of-the-art methods (single agent-based and optimization-based) by a large margin. Computed Tomography (CT), Mangnetic Resonance Imaging (MRI) etc), to align its projections with given 2D Xray images. Reliable 2D/3D registration is a key enabler for image-guided surgeries in modern operating rooms. It brings measurement and plannings done on the pre-operative data into the operating room, and fuse it with intra-operative live 2D X-ray images. It can be used to provide augmented reality image guidance for the surgery, or provide navigation for robotic surgery. Despite that 2D/3D registration has been