Supervised Pix2Pix and unsupervised Cycle-consistency are two modes that dominate the field of medical image-to-image translation. However, neither modes are ideal. The Pix2Pix mode has excellent performance. But it requires paired and well pixel-wise aligned images, which may not always be achievable due to respiratory motion or anatomy change between times that paired images are acquired. The Cycle-consistency mode is less stringent with training data and works well on unpaired or misaligned images. But its performance may not be optimal. In order to break the dilemma of the existing modes, we propose a new unsupervised mode called RegGAN for medical image-to-image translation. It is based on the theory of "loss-correction". In RegGAN, the misaligned target images are considered as noisy labels and the generator is trained with an additional registration network to fit the misaligned noise distribution adaptively. The goal is to search for the common optimal solution to both image-to-image translation and registration tasks. We incorporated RegGAN into a few state-of-the-art image-to-image translation methods and demonstrated that RegGAN could be easily combined with these methods to improve their performances. Such as a simple CycleGAN in our mode surpasses latest NICEGAN even though using less network parameters. Based on our results, RegGAN outperformed both Pix2Pix on aligned data and Cycle-consistency on misaligned or unpaired data. RegGAN is insensitive to noises which makes it a better choice for a wide range of scenarios, especially for medical image-to-image translation tasks in which well pixel-wise aligned data are not available. Code and data used in this study can be found at https://github.com/Kid-Liet/Reg-GAN.

Objective This study aimed to improve the image quality and CT Hounsfield unit accuracy of daily cone-beam computed tomography (CBCT) using registration generative adversarial networks (RegGAN) and apply synthetic CT (sCT) images to dose calculations in radiotherapy. Methods The CBCT/planning CT images of 150 esophageal cancer patients undergoing radiotherapy were used for training (120 patients) and testing (30 patients). An unsupervised deep-learning method, the 2.5D RegGAN model with an adaptively trained registration network, was proposed, through which sCT images were generated. The quality of deep-learning-generated sCT images was quantitatively compared to the reference deformed CT (dCT) image using mean absolute error (MAE), root mean square error (RMSE) of Hounsfield units (HU), and peak signal-to-noise ratio (PSNR). The dose calculation accuracy was further evaluated for esophageal cancer radiotherapy plans, and the same plans were calculated on dCT, CBCT, and sCT images. Results The quality of sCT images produced by RegGAN was significantly improved compared to the original CBCT images. ReGAN achieved image quality in the testing patients with MAE sCT vs. CBCT: 43.7 ± 4.8 vs. 80.1 ± 9.1; RMSE sCT vs. CBCT: 67.2 ± 12.4 vs. 124.2 ± 21.8; and PSNR sCT vs. CBCT: 27.9 ± 5.6 vs. 21.3 ± 4.2. The sCT images generated by the RegGAN model showed superior accuracy on dose calculation, with higher gamma passing rates (93.3 ± 4.4, 90.4 ± 5.2, and 84.3 ± 6.6) compared to original CBCT images (89.6 ± 5.7, 85.7 ± 6.9, and 72.5 ± 12.5) under the criteria of 3 mm/3%, 2 mm/2%, and 1 mm/1%, respectively. Conclusion The proposed deep-learning RegGAN model seems promising for generation of high-quality sCT images from stand-alone thoracic CBCT images in an efficient way and thus has the potential to support CBCT-based esophageal cancer adaptive radiotherapy.

Visual tracking is a challenging problem since it usually faces adverse factors, such as object deformation, fast motion, occlusion, and background clutter in practical applications. Reinforcement learning based Action-Decision Network (ADNet) has shown great potential for object tracking. However, ADNet has some shortcomings in optimal action selection and action reward, and suffers from inefficient tracking. To this end, an improved ADNet is proposed to enhance the tracking accuracy and efficiency. Firstly, the multi-domain training is incorporated into ADNet to further improve the feature extraction ability of its convolution layers. Then, in the reinforcement learning based training phase, both the selection criteria for optimal action and the reward function are redesigned separately to explore more appropriate action and eliminate useless action. Finally, an effective online adaptive update strategy is proposed to adapt to the appearance changes or deformation of the object during actual tracking. Specifically, meta-learning is utilized to pursue the most appropriate parameters for the network so that the parameters are closer to the optimal ones in the subsequent tracking process. Experimental results demonstrate that the proposed tracker has advantages over ADNet in terms of accuracy and efficiency.INDEX TERMS Image processing, visual tracking, reinforcement learning, meta-learning, multi-domain training.

No abstract

Medical images classification is a challenging research topic in the field of computer vision, especially when applied to diagnosis of breast cancer (BC). Nowadays, histopathological image is marked as the gold standard for diagnosing BC. However, such diagnosis is heavily dependent on the clinician's experience, which is extremely time consuming and is subjected to human error even for experienced doctors. To address those problems, this paper implements an automated method for distinguishing the benign from the malignant tumor based on a convolutional neural network (CNN). Traditional deep CNN and machine learning methods not only lead to poor performance, but also fail to make full use of the long-term dependence between some key features and image tags. To further meet the high accuracy requirement of diagnosis, according to the characteristics of histopathological images, we propose a novel CNN framework. Firstly, a normal image is augmented to solve the problem about having a limited database. Secondly, we introduce transfer learning to obtain more accurate weight parameters that were pre-trained on the ImageNet. Thirdly, we combine various features extracted by many individual models to obtain comprehensive features. Finally, random forest is introduced to enforce classification. The experimental results show that novel CNN frameworks have better performance compared with individual models, including DenseNet and ResNet. Experimental results are able to prove the effectiveness of our strategy.