In image captioning, the typical attention mechanisms are arduous to identify the equivalent visual signals especially when predicting highly abstract words. This phenomenon is known as the semantic gap between vision and language. This problem can be overcome by providing semantic attributes that are homologous to language. Thanks to the inherent recurrent nature and gated operating mechanism, Recurrent Neural Network (RNN) and its variants are the dominating architectures in image captioning. However, when designing elaborate attention mechanisms to integrate visual inputs and semantic attributes, RNN-like variants become unflexible due to their complexities. In this paper, we investigate a Transformer-based sequence modeling framework, built only with attention layers and feedforward layers. To bridge the semantic gap, we introduce EnTangled Attention (ETA) that enables the Transformer to exploit semantic and visual information simultaneously. Furthermore, Gated Bilateral Controller (GBC) is proposed to guide the interactions between the multimodal information. We name our model as ETA-Transformer. Remarkably, ETA-Transformer achieves state-of-the-art performance on the MSCOCO image captioning dataset. The ablation studies validate the improvements of our proposed modules.

This paper reviews recent advances in 4D medical imaging (4DMI) and 4D radiation therapy (4DRT), which study, characterize, and minimize patient motion during the processes of imaging and radiotherapy. Patient motion is inevitably present in these processes, producing artifacts and uncertainties in target (lesion) identification, delineation, and localization. 4DMI includes time-resolved volumetric CT, MRI, PET, PET/CT, SPECT, and US imaging. To enhance the performance of these volumetric imaging techniques, parallel multi-detector array has been employed for acquiring image projections and the volumetric image reconstruction has been advanced from the 2D to the 3D tomography paradigm. The time information required for motion characterization in 4D imaging can be obtained either prospectively or retrospectively using respiratory gating or motion tracking techniques. The former acquires snapshot projections for reconstructing a motion-free image. The latter acquires image projections continuously with an associated timestamp indicating respiratory phases using external surrogates and sorts these projections into bins that represent different respiratory phases prior to reconstructing the cyclical series of 3D images. These methodologies generally work for all imaging modalities with variations in detailed implementation. In 4D CT imaging, both multi-slice CT (MSCT) and cone-beam CT (CBCT) are applicable in 4D imaging. In 4D MR imaging, parallel imaging with multi-coil-detectors has made 4D volumetric MRI possible. In 4D PET and SPECT, rigid and non-rigid motions can be corrected with aid of rigid and deformable registration, respectively, without suffering from low statistics due to signal binning. In 4D PET/CT and SPECT/CT, a single set of 4D images can be utilized for motion-free image creation, intrinsic registration, and attenuation correction. In 4D US, volumetric ultrasonography can be employed to monitor fetal heart beating with relatively high temporal resolution. 4DRT aims to track and compensate for target motion during radiation treatment, minimizing normal tissue injury, especially critical structures adjacent to the target, and/or maximizing radiation dose to the target. 4DRT requires 4DMI, 4D radiation treatment planning (4D RTP), and 4D radiation treatment delivery (4D RTD). Many concepts in 4DRT are borrowed, adapted and extended from existing image-guided radiation therapy (IGRT) and adaptive radiation therapy (ART). The advantage of 4DRT is its promise of sparing additional normal tissue by synchronizing the radiation beam with the moving target in real-time. 4DRT can be implemented differently depending upon how the time information is incorporated and utilized. In an ideal situation, the motion adaptive approach guided by 4D imaging should be applied to both RTP and RTD. However, until new automatic planning and motion feedback tools are developed for 4DRT, clinical implementation of ideal 4DRT will meet with limited success. However, simplified forms of 4DRT have been implemented with minor mo...

Withthe technological advent, the clustering phenomenon is recently being used in various domains and in natural language recognition. This article contributes to the clustering phenomenon of natural language and fulfills the requirements for the dynamic update of the knowledge system. This article proposes a method of dynamic knowledge extraction based on sentence clustering recognition using a neural network-based framework. The conversion process from natural language papers to object-oriented knowledge system is studied considering the related problems of sentence vectorization. This article studies the attributes of sentence vectorization using various basic definitions, judgment theorem, and postprocessing elements. The sentence clustering recognition method of the network uses the concept of prereliability as a measure of the credibility of sentence recognition results. An ART2 neural network simulation program is written using MATLAB, and the effect of the neural network on sentence recognition is utilized for the corresponding analysis. A postreliability evaluation indexing is done for the credibility of the model construction, and the implementation steps for the conjunctive rule sentence pattern are specifically introduced. A new method of structural modeling is utilized to generate the structured derivation relationship, thus completing the natural language knowledge extraction process of the object-oriented knowledge system. An application example with mechanical CAD is used in this work to demonstrate the specific implementation of the example, which confirms the effectiveness of the proposed method.

Purpose To compare the image quality of amplitude-binned four-dimensional magnetic resonance imaging (4DMRI) reconstructed using two concurrent respiratory (navigator and bellows) waveforms. Methods and Materials A prospective, respiratory-correlated 4DMRI scanning program was employed to acquire T2-weighted single-breath 4DMRI image using an internal navigator and external bellows. After 10-second training of a surrogate signal, 2D MRI image acquisition was triggered at a level (bin) and anatomic location (slice) until the bin-slice table was filled for 4DMRI reconstruction. The bellows signal was always collected, even when the navigator trigger was used, to retrospectively reconstruct a bellows-rebinned 4DMRI. Ten volunteers participated in this IRB-approved 4DMRI study and four scans were acquired for each subject, including coronal and sagittal scans triggered by either navigator or bellows, and six 4DMRI images (navigator-triggered, bellows-rebinned, and bellows-triggered) were reconstructed. The simultaneously acquired waveforms and resulting 4DMRI image quality were compared using signal correlation, bin/phase shift, and binning motion artifacts. The consecutive bellows-triggered 4DMRI was used for indirect comparison. Results Correlation coefficients between navigator and bellows signals were found to be patient-specific and inhalation-/exhalation-dependent, ranging from 0.1 to 0.9, due to breathing irregularities (>50% scans) and commonly-observed bin/phase shifts (−1.1±0.6 bin) in both 1D waveforms and diaphragm motion extracted from 4D images. The navigator-triggered 4DMRI contains much fewer binning motion artifacts at the diaphragm than bellows-rebinned and bellows-triggered 4DMRI. Coronal scans are faster than sagittal scans due to fewer slices and higher achievable acceleration factors. Conclusion Navigator-triggered 4DMRI contains substantially fewer binning motion artifacts than bellows-rebinned and bellows-triggered 4DMRI, primarily due to the deviation of the external from the internal surrogate. This study compares two concurrent surrogates during the same 4DMRI scan and their resulting 4DMRI quality. The navigator-triggered 4DMRI scanning protocol is preferred to the bellows-based, especially the coronal scans, for clinical respiratory motion simulation.

In the last six years, the Developmental Therapeutics Program (DTP) of the US National Cancer Institute (NCI) has screened over 60,000 chemical compounds and a larger number of natural product extracts for their ability to inhibit growth of 60 different cancer cell lines representing different organs of origin. Whereas inhibition of the growth of one cancer cell type gives no information on drug specificity, the relative growth inhibitory activities against 60 different cells constitute patterns that encode detailed information on mechanisms of action and resistance (as reviewed in Boyd and Paull, Drug Devel. Res. 1995, 34, 19-109 and Weinstein et al., Science 1997, 275, 343-349). In order to correlate the patterns of activity with properties of the cells, we and other laboratories are characterizing the cells with respect to a large number of factors at the DNA, mRNA, and protein levels. As part of that effort, we have developed a two-dimensional gel electrophoresis (2-DE) protein expression database covering all 60 cell types (Buolamwini et al., submitted). Here we present analyses of the correlations among protein spots (i) in terms of their patterns of expression and (ii) in terms of their apparent relationships to the pharmacology of a set of 3989 screened compounds. The correlations tend to be stronger for the latter than for the former, suggesting that the spots have more robust signatures in terms of the pharmacology than in terms of expression levels. Links to pertinent databases and tools of analysis will be updated progressively at http:@www.nci.nih.gov/intra/lmp/jnwbio.htm and http:@epnwsl.ncifcrf.gov:2345/dis3d/dtp.++ +html.