D-CNN: A New model for Generating Image Captions with Text Extraction Using Deep Learning for Visually Challenged Individuals

Abstract

Automatically describing the information of an image using properly constructed sentences is a tricky task in any language. However, it has the potential to have a significant effect by enabling visually challenged individuals to better understand their surroundings. This paper proposes an image captioning system that generates detailed captions and extracts text from an image, if any, and uses it as a part of the caption to provide a more precise description of the image. To extract the image features, the proposed model uses Convolutional Neural Networks (CNNs) followed by Long Short-Term Memory (LSTM) that generates corresponding sentences based on the learned image features. Further, using the text extraction module, the extracted text (if any) is included in the image description and the captions are presented in audio form. Publicly available benchmark datasets for image captioning like MS COCO, Flickr-8k, Flickr-30k have a variety of images, but they hardly have images that contain textual information. These datasets are not sufficient for the proposed model and this has resulted in the creation of a new image caption dataset that contains images with textual content. With the newly created dataset, comparative analysis of the experimental results is performed on the proposed model and the existing pre-trained model. The obtained experimental results show that the proposed model is equally effective as the existing one in subtitle image captioning models and provides more insights about the image by performing text extraction.

The New Dataset MITWPU-1K for Object Recognition and Image Captioning Tasks

“…The dataset contains some images which contain text in the form of a banner or poster as shown in Figure 12. Using this dataset, we proposed a new deep learning model [10] for performing image captioning and text extraction. The obtained results of image captioning, including textual information present in the image, were satisfactory, giving accuracy up to 83% and performed as well as the state of the art methods.…”

Red Deer Optimization with Artificial Intelligence Enabled Image Captioning System for Visually Impaired People

“…Besides, the generated textual sentence can be transformed into speech. Bhalekar et al [12] devise an image captioning mechanism that produces comprehensive captions, derives text from imagery, if any, and utilizes it as a part of the caption to offer a highly accurate description of the imagery. The devised method will use LSTM and CNNs to extract the image features to produce respective sentences related to the learned image features.…”

Substation Danger Sign Detection and Recognition using Convolutional Neural Networks

“…They can also accurately localize a portion of an image or data cluster using tools such as regression. In our efforts to deploy a model of CNN compatible with image extraction, we took inspiration from literature such as [14][15][16]. The main methodology applied in recent literature for detection networks are:  Direct detection [14,15]  Regional detection [13,17]…”