Art2Real: Unfolding the Reality of Artworks via Semantically-Aware Image-To-Image Translation

Tomei, Matteo; Cornia, Marcella; Baraldi, Lorenzo; Cucchiara, Rita

doi:10.1109/cvpr.2019.00600

Cited by 75 publications

(38 citation statements)

References 46 publications

(87 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Chang et al [79] proposed an image-to-image translation approach called Symparameterized Generative Network (SGN), and it focuses on the loss area and infers translations of images in mixed domains by learning the combined characteristics of each domain. Tomei et al [80] proposed a semantic-aware approach that can reduce the gap between visual features of artistic and realistic data by translating artworks to photo-realistic visualizations. Mo et al [81] proposed an unsupervised image-to-image translation approach called instance-aware GAN (InstaGAN), which can not only incorporate the instance information but also improve the multiinstance transfiguration.…”

Section: ) Other Methodsmentioning

confidence: 99%

A State-of-the-Art Review on Image Synthesis With Generative Adversarial Networks

et al. 2020

View full text Add to dashboard Cite

Generative Adversarial Networks (GANs) have achieved impressive results in various image synthesis tasks, and are becoming a hot topic in computer vision research because of the impressive performance they achieved in various applications. In this paper, we introduce the recent research on GANs in the field of image processing, including image synthesis, image generation, image semantic editing, image-to-image translation, image super-resolution, image inpainting, and cartoon generation. We analyze and summarize the methods used in these applications which have improved the generated results. Then, we discuss the challenges faced by GANs and introduce some methods to deal with these problems. We also preview some likely future research directions in the field of GANs, such as video generation, facial animation synthesis and 3D face reconstruction. The purpose of this review is to provide insights into the research on GANs and to present the various applications based on GANs in different scenarios. INDEX TERMS Generative adversarial networks, image synthesis, image-to-image translation, image editing, cartoon generation.

show abstract

Section: ) Other Methodsmentioning

confidence: 99%

A State-of-the-Art Review on Image Synthesis With Generative Adversarial Networks

et al. 2020

View full text Add to dashboard Cite

show abstract

“…In the last years, several efforts have been done to apply computer vision techniques to the cultural heritage domain resulting in different works and applications ranging from generative models to classification and retrieval solutions. On the generative and synthesis side, up-and-coming results have been obtained by style transfer models that aim to transfer the style of a painting to a real photo [9] and, on the contrary, create a realistic representation of a given painting [23,24]. On a different note, several large-scale art datasets have been proposed to foster researches on this domain, with a particular focus on style and genre recognition [12,18].…”

Section: Computer Vision For Cultural Heritagementioning

confidence: 99%

Artpedia: A New Visual-Semantic Dataset with Visual and Contextual Sentences in the Artistic Domain

Stefanini

Cornia

Baraldi

et al. 2019

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

As vision and language techniques are widely applied to realistic images, there is a growing interest in designing visual-semantic models suitable for more complex and challenging scenarios. In this paper, we address the problem of cross-modal retrieval of images and sentences coming from the artistic domain. To this aim, we collect and manually annotate the Artpedia dataset that contains paintings and textual sentences describing both the visual content of the paintings and other contextual information. Thus, the problem is not only to match images and sentences, but also to identify which sentences actually describe the visual content of a given image. To this end, we devise a visual-semantic model that jointly addresses these two challenges by exploiting the latent alignment between visual and textual chunks. Experimental evaluations, obtained by comparing our model to different baselines, demonstrate the effectiveness of our solution and highlight the challenges of the proposed dataset. The Artpedia dataset is publicly available at: http://aimagelab.ing.unimore.it/artpedia.

show abstract

“…Following this line, in a preliminary work [24] we reached better results with respect to the Cycle-GAN baseline. Later, we further improved the realism of the generation by considering patches as members of specific semantic classes and trying to preserve this membership during the generation [25]. Memory banks.…”

Section: Semantic-consistency and Realistic Detailsmentioning

confidence: 99%

“…It shall be noted, on the other side, that it is not feasible to re-train state-of-the-art architectures on artistic data, as no large annotated datasets exist in the cultural heritage domain. To address this domain-shift problem while still exploiting the knowledge learned in pre-trained architectures, we have recently proposed a pixel-level domain translation architecture [25], that can map paintings to photo-realistic visualizations by generating translation images which look realistic while preserving the semantic content of the painting. The problem is one of unpaired domain translation, as no annotated pairing exists, i.e.…”

Section: Introductionmentioning

confidence: 99%

Image-to-Image Translation to Unfold the Reality of Artworks: An Empirical Analysis

Tomei

Cornia

Baraldi

et al. 2019

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

0000−0002−1385−924X] , Marcella Cornia [0000−0001−9640−9385] , Lorenzo Baraldi [0000−0001−5125−4957] , and Rita Cucchiara [0000−0002−2239−283X]Abstract. State-of-the-art Computer Vision pipelines show poor performances on artworks and data coming from the artistic domain, thus limiting the applicability of current architectures to the automatic understanding of the cultural heritage. This is mainly due to the difference in texture and low-level feature distribution between artistic and real images, on which state-of-the-art approaches are usually trained. To enhance the applicability of pre-trained architectures on artistic data, we have recently proposed an unpaired domain translation approach which can translate artworks to photo-realistic visualizations. Our approach leverages semantically-aware memory banks of real patches, which are used to drive the generation of the translated image while improving its realism. In this paper, we provide additional analyses and experimental results which demonstrate the effectiveness of our approach. In particular, we evaluate the quality of generated results in the case of the translation of landscapes, portraits and of paintings coming from four different styles using automatic distance metrics. Also, we analyze the response of pre-trained architecture for classification, detection and segmentation both in terms of feature distribution and entropy of prediction, and show that our approach effectively reduces the domain shift of paintings. As an additional contribution, we also provide a qualitative analysis of the reduction of the domain shift for detection, segmentation and image captioning.

show abstract

Art2Real: Unfolding the Reality of Artworks via Semantically-Aware Image-To-Image Translation

Cited by 75 publications

References 46 publications

A State-of-the-Art Review on Image Synthesis With Generative Adversarial Networks

A State-of-the-Art Review on Image Synthesis With Generative Adversarial Networks

Artpedia: A New Visual-Semantic Dataset with Visual and Contextual Sentences in the Artistic Domain

Image-to-Image Translation to Unfold the Reality of Artworks: An Empirical Analysis

Contact Info

Product

Resources

About