MSCap: Multi-Style Image Captioning With Unpaired Stylized Text

Guo, Longteng; Liu, Jing; Yao, Peng; Li, Jiangwei; Lu, Hanqing

doi:10.1109/cvpr.2019.00433

Cited by 112 publications

(85 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some recent image captioning studies [26,21,31] have constructed variant LSTM language models to learn factual and non-factual knowledge in corpora. Some studies [32,21,33,34] have allowed for learning non-factual knowledge in unpaired corpora via weakly supervised or unsupervised methods. These methods are expected (but not limited) to be used to train better models for autogenerating factual and functional captions of drug paraphernalia.…”

Section: Discussionmentioning

confidence: 99%

“…Chen et al [32] proposed generating non-factual image captions with the Domain Layer Norm, which enables the generation of various stylized sentences. MSCap [33] was designed to generate multiple stylized descriptions by training a single captioning model on an unpaired non-factual corpus with the help of several auxiliary modules. Zhao et al [34] proposed a new model named MemCap, which resorts to explicitly encoding non-factual knowledge by building a memory module.…”

Section: Related Work a Image Captioning Datasetsmentioning

confidence: 99%

See 1 more Smart Citation

DrunaliaCap: Image Captioning for Drug-Related Paraphernalia With Deep Learning

Zhao

2020

IEEE Access

View full text Add to dashboard Cite

Image captioning is a process of generating textual descriptions of images. In recent years, research on publicly available large-scale datasets and deep learning-based algorithms has promoted the development of this field. However, little research has been conducted on captioning images of drug-related paraphernalia that, despite being an important topic for both drug prevention and police enforcement, is not covered by existing image captioning studies. In this paper, we propose DrunaliaCap-a deep learningbased system for autogenerating both "factual" (what is in the image) and "functional" (the usage of each paraphernalia during drug-taking) descriptions of images of drug-related paraphernalia. We constructed a new dataset containing 20 categories of drug-related items and trained deep learning-based models for the proposed system. We further proposed a method to evaluate and optimize the generation of captions to prevent them from missing important knowledge. Experiments were conducted to validate the performance of the newly proposed dataset and method. We analyzed the experimental results and discussed the significance, limitations, and potential applications of our work. INDEX TERMS image captioning, drug prevention, dataset construction, deep learning VOLUME 4, 2016 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Related Work a Image Captioning Datasetsmentioning

confidence: 99%

DrunaliaCap: Image Captioning for Drug-Related Paraphernalia With Deep Learning

Zhao

2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Thus, methods developed on such datasets might not be easily adopted in the wild. Nevertheless, great efforts have been made to extend captioning to out-of-domain data [3,9,69] or different styles beyond mere factual descriptions [22,55]. In this work we explore unsupervised captioning, where image and language sources are independent.…”

Section: Language Domainmentioning

confidence: 99%

“…In [69], the cross-domain problem is addressed with a cycle objective. Similarly, unpaired data can be used to generate stylized descriptions [22,46]. Anderson et al [3] propose a method to complete partial sequence data, e.g.…”

Section: Related Workmentioning

confidence: 99%

Towards Unsupervised Image Captioning With Shared Multimodal Embeddings

Laina

Rupprecht

Navab

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Understanding images without explicit supervision has become an important problem in computer vision. In this paper, we address image captioning by generating language descriptions of scenes without learning from annotated pairs of images and their captions. The core component of our approach is a shared latent space that is structured by visual concepts. In this space, the two modalities should be indistinguishable. A language model is first trained to encode sentences into semantically structured embeddings. Image features that are translated into this embedding space can be decoded into descriptions through the same language model, similarly to sentence embeddings. This translation is learned from weakly paired images and text using a loss robust to noisy assignments and a conditional adversarial component. Our approach allows to exploit large text corpora outside the annotated distributions of image/caption data. Our experiments show that the proposed domain alignment learns a semantically meaningful representation which outperforms previous work.

show abstract

“…Based on the encoderdecoder pipeline [24,30,33], much progress has been made on image captioning. For example, [27] introduces the visual attention that adaptively attends to the salient areas in the image, [18] proposes an adaptive attention model that decides whether to attend to the image or to the visual sentinel, [29] corporates learned language bias as a language prior for more human-like captions, [19] and [8] focus on the discriminability and style properties of image captions respectively, and [22] adopts reinforcement learning (RL) that directly optimize evaluation metric.…”

Section: Related Workmentioning

confidence: 99%

Aligning Linguistic Words and Visual Semantic Units for Image Captioning

Guo

Liu

Tang

et al. 2019

Proceedings of the 27th ACM International Conference on Multimedia

Self Cite

100

View full text Add to dashboard Cite

Image captioning attempts to generate a sentence composed of several linguistic words, which are used to describe objects, attributes, and interactions in an image, denoted as visual semantic units in this paper. Based on this view, we propose to explicitly model the object interactions in semantics and geometry based on Graph Convolutional Networks (GCNs), and fully exploit the alignment between linguistic words and visual semantic units for image captioning. Particularly, we construct a semantic graph and a geometry graph, where each node corresponds to a visual semantic unit, i.e., an object, an attribute, or a semantic (geometrical) interaction between two objects. Accordingly, the semantic (geometrical) context-aware embeddings for each unit are obtained through the corresponding GCN learning processers. At each time step, a context gated attention module takes as inputs the embeddings of the visual semantic units and hierarchically align the current word with these units by first deciding which type of visual semantic unit (object, attribute, or interaction) the current word is about, and then finding the most correlated visual semantic units under this type. Extensive experiments are conducted on the challenging MS-COCO image captioning dataset, and superior results are reported when comparing to state-of-the-art approaches. The code is publicly available at https://github.com/ltguo19/VSUA-Captioning.

show abstract

MSCap: Multi-Style Image Captioning With Unpaired Stylized Text

Cited by 112 publications

References 20 publications

DrunaliaCap: Image Captioning for Drug-Related Paraphernalia With Deep Learning

DrunaliaCap: Image Captioning for Drug-Related Paraphernalia With Deep Learning

Towards Unsupervised Image Captioning With Shared Multimodal Embeddings

Aligning Linguistic Words and Visual Semantic Units for Image Captioning

Contact Info

Product

Resources

About