An Interactive Image Editing System Using an Uncertainty-Based Confirmation Strategy

Abstract

We propose an interactive image editing system that has a confirmation dialogue strategy using an entropy-based uncertainty calculation on its generated images with Deep Convolutional Generative Adversarial Networks (DCGAN). DCGAN is an image generative model that learns an image manifold of a given dataset and enables continuous change of an image. Our proposed image editing system combines DCGAN with a natural language interface that accepts image editing requests in natural language. Although such a system is helpful for human users, it often faces uncertain requests to generate acceptable images. A promising approach to solve this problem is introducing a dialogue process that shows multiple candidates and confirms the user's intention. However, confirming every editing request creates redundant dialogues. To achieve more efficient dialogues, we propose an entropy-based dialogue strategy that decides when the system should confirm, and enables effective image editing through a dialogue that reduces redundant confirmations. We conducted image editing dialogue experiments using an avatar face illustration dataset for editing by natural language requests. Through quantitative and qualitative analysis, our results show that our entropy-based confirmation strategy achieved an effective dialogue by generating images desired by users. INDEX TERMS Confirmation, generative adversarial networks, image editing, natural language interface.

Text as Neural Operator:Image Manipulation by Text Instruction

“…The research in this area aims to manipulate image content in a controlled manner. To enable user-guided manipulation, a variety of frameworks [3,9,21,23,33,37,43,47,52,57,62,63,70,75,77,78,81] have been proposed to study different control signals. For instance, Zhang et al [78] and Zou et al [86] used sparse dots and text respectively to guide the image colorization process.…”