Self-Play Reinforcement Learning for Fast Image Retargeting

Abstract

In this study, we address image retargeting, which is a task that adjusts input images to arbitrary sizes. In one of the best-performing methods called MULTIOP, multiple retargeting operators were combined and retargeted images at each stage were generated to find the optimal sequence of operators that minimized the distance between original and retargeted images. The limitation of this method is in its tremendous processing time, which severely prohibits its practical use. Therefore, the purpose of this study is to find the optimal combination of operators within a reasonable processing time; we propose a method of predicting the optimal operator for each step using a reinforcement learning agent. The technical contributions of this study are as follows. Firstly, we propose a reward based on self-play, which will be insensitive to the large variance in the content-dependent distance measured in MULTIOP. Secondly, we propose to dynamically change the loss weight for each action to prevent the algorithm from falling into a local optimum and from choosing only the most frequently used operator in its training. Our experiments showed that we achieved multi-operator image retargeting with less processing time by three orders of magnitude and the same quality as the original multi-operator-based method, which was the best-performing algorithm in retargeting tasks. CCS CONCEPTS • Computing methodologies → Reinforcement learning; Image processing; Neural networks.

Fast Hybrid Image Retargeting

“…Image size Processor Type CycleIR [6] 370 1024×768 GPU BEL [9] 1,000 615×461 CPU Ref. [5] 9,900 640×480 CPU Ref. [7] 120,000 640×480 N/A Ref.…”

“…Continuous methods change image dimensions by finding a mapping (a warp) from source to target images [13,14,15]. More recently, a variety of retargeting algorithms [6,7,8,9,10,3,16,17,18] and on reinforcement learning [5] have been proposed. In this section, we describe prior art which is of particular relevance to our method.…”

“…Our implementation has been tested both on desktop and on mobile devices, and a user study based on the Re-targetMe benchmark [4] shows that our method outperforms The authors wish to thank Mete Ozay for his advice and support. recent approaches, while executing in a fraction of the time [5,6,7,8,9,10].…”

“…The authors wish to thank Mete Ozay for his advice and support. recent approaches, while executing in a fraction of the time [5,6,7,8,9,10].…”

Image resizing by reconstruction from deep features

“…However, human intervention is required. Kajiura et al [27] used reinforcement learning to find the optimal retargeting order when using a multi-operator approach. In other work, Esmaeili et al [28] proposed an automatic thumbnail generation network that does not utilize a saliency map.…”