Effective Pre-Training Method and Its Compositional Intelligence for Image Captioning

Choi, Won-Hyuk; Choi, Yong-Suk

doi:10.3390/s22093433

Cited by 2 publications

(2 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our best model was trained with the capsule network and inception-V3 as a feature extractor, with caption enrichment by an external contextual description. The results are the basis for future research that will generate more conceptual and specific descriptions by considering emotions in captions and using transformers in the decoder since this network have extraordinary performance in image captioning [ 54 ].…”

Section: Discussionmentioning

confidence: 99%

Caps Captioning: A Modern Image Captioning Approach Based on Improved Capsule Network

Javanmardi

Latif

Sadeghi

et al. 2022

Sensors

View full text Add to dashboard Cite

In image captioning models, the main challenge in describing an image is identifying all the objects by precisely considering the relationships between the objects and producing various captions. Over the past few years, many methods have been proposed, from an attribute-to-attribute comparison approach to handling issues related to semantics and their relationships. Despite the improvements, the existing techniques suffer from inadequate positional and geometrical attributes concepts. The reason is that most of the abovementioned approaches depend on Convolutional Neural Networks (CNNs) for object detection. CNN is notorious for failing to detect equivariance and rotational invariance in objects. Moreover, the pooling layers in CNNs cause valuable information to be lost. Inspired by the recent successful approaches, this paper introduces a novel framework for extracting meaningful descriptions based on a parallelized capsule network that describes the content of images through a high level of understanding of the semantic contents of an image. The main contribution of this paper is proposing a new method that not only overrides the limitations of CNNs but also generates descriptions with a wide variety of words by using Wikipedia. In our framework, capsules focus on the generation of meaningful descriptions with more detailed spatial and geometrical attributes for a given set of images by considering the position of the entities as well as their relationships. Qualitative experiments on the benchmark dataset MS-COCO show that our framework outperforms state-of-the-art image captioning models when describing the semantic content of the images.

show abstract

Section: Discussionmentioning

confidence: 99%

Caps Captioning: A Modern Image Captioning Approach Based on Improved Capsule Network

Javanmardi

Latif

Sadeghi

et al. 2022

Sensors

View full text Add to dashboard Cite

show abstract

“…These pre-trained models have been trained on large and diverse datasets and learned highly discriminative features useful for a wide range of image classification tasks. By reusing pre-trained models, we can reduce the computational cost and training time required for a new task and achieve improved performance [18], especially in the medical domain where large, diverse datasets are scarce.…”

Section: Introductionmentioning

confidence: 99%

Multi-Input Deep Learning Approach for Breast Cancer Screening Using Thermal Infrared Imaging and Clinical Data

et al. 2023

View full text Add to dashboard Cite

Breast cancer is one of the most prevalent causes of death among women across the globe. Early detection is the best strategy for reducing the mortality rate. Currently, mammography is the standard screening modality, which has its shortcomings. To complement this modality, thermal infrared-based Computer-Aided Diagnosis (CADx) tools have been presented as economical, less hazardous, and a suitable solution for various age groups. Although a viable solution, most CADx systems are built primarily from frontal breast thermograms, and are likely to miss lesions that may develop on the sides. Additionally, these systems often disregard critical clinical data, such as risk factors. This paper presents a novel CADx system that utilizes deep learning techniques for breast cancer detection. The system incorporates multiple breast thermogram views and corresponding patient clinical data to improve the accuracy of the diagnosis. We describe the methodology of the system, including the extraction of regions of interest from images and the use of transfer learning to train three different models. We evaluate the performance of the models and compare them to similar works from the literature. The results demonstrate that using multi-inputs outperforms single-input models and achieves an overall accuracy of 90.48%, a sensitivity of 93.33%, and an AUROC curve of 0.94. This approach could offer a more cost-effective and less hazardous screening option for breast cancer detection, particularly for a wide range of age groups.

show abstract

Effective Pre-Training Method and Its Compositional Intelligence for Image Captioning

Cited by 2 publications

References 41 publications

Caps Captioning: A Modern Image Captioning Approach Based on Improved Capsule Network

Caps Captioning: A Modern Image Captioning Approach Based on Improved Capsule Network

Multi-Input Deep Learning Approach for Breast Cancer Screening Using Thermal Infrared Imaging and Clinical Data

Contact Info

Product

Resources

About