Focusing on your subject: Deep subject-aware image composition recommendation networks

Abstract

Photo composition is one of the most important factors in the aesthetics of photographs. As a popular application, composition recommendation for a photo focusing on a specific subject has been ignored by recent deep-learning-based composition recommendation approaches. In this paper, we propose a subject-aware image composition recommendation method, SAC-Net, which takes an RGB image and a binary subject window mask as input, and returns good compositions as crops containing the subject. Our model first determines candidate scores for all possible coarse cropping windows. The crops with high candidate scores are selected and further refined by regressing their corner points to generate the output recommended cropping windows. The final scores of the refined crops are predicted by a final score regression module. Unlike existing methods that need to preset several cropping windows, our network is able to automatically regress cropping windows with arbitrary aspect ratios and sizes. We propose novel stability losses for maximizing smoothness when changing cropping windows along with view changes. Experimental results show that our method outperforms state-of-the-art methods not only on the subject-aware image composition recommendation task, but also for general purpose composition recommendation. We also have designed a multistage labeling scheme so that a large amount of ranked pairs can be produced economically. We use this scheme to propose the first subject-aware composition dataset SACD, which contains 2777 images, and more than 5 million composition ranked pairs. The SACD dataset is publicly available at https://cg.cs.tsinghua.edu.cn/SACD/.

Learning Subject-Aware Cropping by Outpainting Professional Photos

“…Only 1.1K, 339, 176, and 39 of the images in CPC, GAICD, FCDB and FLMS (Fang et al 2014) are human-centric (Zhang et al 2022); the human-centric evaluation sets consist of only 50 images from GAICD and 215 from FLMS and FCDB combined. SACD (Yang et al 2023) is a recent dataset for subject-aware cropping that does not focus on a particular subject type but contains 24K+ labels and 5.2 million ranking pairs generated using their annotation procedure. Our approach differs in that we generate a dataset to provide weak-supervision for a given subject type.…”

“…Prior cropping datasets such as FLMS (Fang et al 2014), FCDB (Chen et al 2017a), and SACD (Yang et al 2023) lack the quantity of images in any particular subject category needed to serve as evaluation (having only 500, 348, and 290 test images total). In order to evaluate GenCrop, we construct new evaluation sets for the six aforementioned subjects, derived from the Unsplash testing images.…”

“…In order to evaluate GenCrop, we construct new evaluation sets for the six aforementioned subjects, derived from the Unsplash testing images. We select Unsplash images with space for tighter crops (alternative framings) and task the model to produce crops that pre- (Zhang et al 2022) and (Yang et al 2023), respectively. FCDB+FLMS is the human-centric test-set used by (Zhang et al 2022).…”

“…Framing and cropping decide what elements of a scene to include or exclude from the image, and how to frame or crop an image is influenced by the subject that one wishes to portray. Subjectaware cropping takes a notion of a subject in addition to pixels and has been studied in recent work on data-driven approaches (Yang et al 2023) and in the context of humancentric images (Zhang et al 2022). High-quality solutions to this problem are based on supervised learning, from large datasets of manual annotations created specifically for cropping (Zeng et al 2019;Wei et al 2018;Yang et al 2023).…”

“…To demonstrate the effectiveness of GenCrop, we evaluate it on the human-centric (portrait) and the subject-aware cropping tasks proposed by (Zhang et al 2022) and (Yang et al 2023). We show that GenCrop yields competitive results against fully supervised approaches (Zhang et al 2022) on the existing datasets (Fang et al 2014;Chen et al 2017a;Yang et al 2023) (under quantitative metrics such as Intersection-over-Union and boundary displacement (Zhang et al 2022)), while being superior to the best weakly/unsupervised method (Chen et al 2017b). We also evaluate Gen-Crop on additional subject categories such as cats, dogs, etc.…”