Deep Interactive Region Segmentation and Captioning

Abstract

With recent innovations in dense image captioning, it is now possible to describe every object of the scene with a caption while objects are determined by bounding boxes. However, interpretation of such an output is not trivial due to the existence of many overlapping bounding boxes. Furthermore, in current captioning frameworks, the user is not able to involve personal preferences to exclude out of interest areas. In this paper, we propose a novel hybrid deep learning architecture for interactive region segmentation and captioning where the user is able to specify an arbitrary region of the image that should be processed. To this end, a dedicated Fully Convolutional Network (FCN) named Lyncean FCN (LFCN) is trained using our special training data to isolate the User Intention Region (UIR) as the output of an efficient segmentation. In parallel, a dense image captioning model is utilized to provide a wide variety of captions for that region. Then, the UIR will be explained with the caption of the best match bounding box. To the best of our knowledge, this is the first work that provides such a comprehensive output. Our experiments show the superiority of the proposed approach over state-of-the-art interactive segmentation methods on several well-known datasets. In addition, replacement of the bounding boxes with the result of the interactive segmentation leads to a better understanding of the dense image captioning output as well as accuracy enhancement for the object detection in terms of Intersection over Union (IoU).

A fully convolutional two-stream fusion network for interactive image segmentation

“…A fully convolutional network (FCN) is trained to predict the foreground/background from image-user interaction pairs. With similar imageuser interaction pairs as input to the network, Boroujerdi et al [17] use a lyncean fully convolutional network to predict foreground/background. This network replaces the last two convolutional layers in the FCN in [15] with three convolutional layers with gradually decreased kernel size to better capture the geometry of objects.…”

“…This algorithm leads to improved performance, since it is more closely aligned with the patterns of real users. Essentially, all these networks [15,17,16,18,19,21,28] adopt early fusion structures. They combine the image and the user interaction features from the first layer of DNN.…”

“…Recently, deep features produced by deep neural networks (DNNs) have shown their power in many computer vision tasks including image classification [8,9,10] and semantic segmentation [11,12,13,14]. Thus, several researchers [15,16,17,18,19,20,21] have used DNNs to extract deep features with higher-level understanding for image and user interactions to improve interactive image segmentation. Most of these DNN-based methods can be viewed as an early fusion of features using DNN.…”

Block Annotation: Better Image Annotation With Sub-Image Decomposition

“…In these cases the cost of pixel-level annotation can be reduced by automating a portion of the task. Matting and object selection [50,33,34,6,58,57,10,30,59] generate tight boundaries from loosely annotated boundaries or few inside/outside clicks and scribbles. [44,38] introduced a predictive method which automatically infers a foreground mask from 4 boundary clicks, and was extended to full-image segmentation in [2].…”