Image Captioning With Positional and Geometrical Semantics

Abstract

The last 5 to 6 years have seen tremendous progress in automatic image captioning using deep learning. Initial research focused on the attribute-to-attribute comparison of image features and texts to describe the image as a sentence, the current research is handling issues related to semantics and correlations. However, current state of art research suffers from insufficient concepts when it comes to positional and geometrical attributes. The majority of research relying on CNN's (Convolutional Neural Networks) for object feature extractions has no clue about equivariance and rotational invariance which leads towards the orientation-less understanding of objects for captioning along with longer training time, and larger dataset. Furthermore, CNN's based image captioning encoders also fail to understand the geometrical alignment of object attributes within the image and hence mislabels distorted as correct. To cater to the above issues, we propose ICPS (Captioning with Positional and geometrical Semantics) a capsule network-based image captioning technique along with transformer neural networks as the decoder. The proposed ICPS architecture handles various geometrical properties of image objects with the help of parallelized capsules while the object-to-text decoding is done by Transformer Neural Networks. The inclusion of cluster capsules provides better object understanding in terms of position, equivariance, and geometrical orientation with more augmented object understanding over a small dataset in comparatively less time. The extracted image features provide a better understanding of image objects and help the decoding stage to narrate effectively with positional and geometrical details. We trained and tested our ICPS over the Flickr8k dataset and found ourselves to be better at captioning when it comes to describing the positional and geometrical transitions as compared to other current state-of-the-art research.

Pneumonia classification: A limited data approach for global understanding

“…Our approach leverages a pre-trained Capsule Networks Cluster [ 28 ], initially trained on the Flickr dataset for generating image captions with detailed positional and geometrical information. We have customized this pre-trained model to extract inflammation-specific features within the inflammation capsule layer.…”

“…In our work, we aim to aid in diagnosing inflammation due to pneumonia in chest X-rays by providing a lightweight geometrical and positional understanding-based deep learning network. Our proposed network is a composition of a capsule networks cluster [ 28 ] and a modified class activation map. The capsule networks cluster-based model is light because it requires very little data to be trained to understand the geometrical, orientational, and positional inflammation details in CXR.…”

Pneumonia Classification: A limited data approach for global understanding

“…Our methodology consumes a pre-trained Capsule Network Cluster [28]. The capsule network cluster was trained over the flicker dataset and has been used for generating captions against images with positional and geometrical information.…”

PerceptGuide: A Perception Driven Assistive Mobility Aid Based on Self-Attention and Multi-Scale Feature Fusion

“…Haque et al presented an approach to mimic human using attention and object features [53]. A different approach with geometrical semantics to generate captions was introduced in [54].…”