Human-Centric Image Cropping with Partition-Aware and Content-Preserving Features

Learning Subject-Aware Cropping by Outpainting Professional Photos

“…Data-driven approaches for cropping. Most recent works (Hong et al 2021;Zeng et al 2019;Zhang et al 2022;Jia et al 2022;Pan et al 2021;Wang et al 2023) utilize direct supervision and require large amounts of humanannotated crops and crop scores to train. GAIC (Zeng et al 2019), CPC (Wei et al 2018), and FCDB (Chen et al 2017a) are commonly used datasets.…”

“…These datasets are expensive to annotate (CPC and GAIC have 259K and 106K crops; on average 24 and 86 crops per image) and the quality of these images and crowd-sourced crops can vary (see supplemental §B.3). Since these datasets are not specific to a subject type (e.g., human), prior work on human-centric cropping (Zhang et al 2022) is limited by the number of images available for training and evaluation. Only 1.1K, 339, 176, and 39 of the images in CPC, GAICD, FCDB and FLMS (Fang et al 2014) are human-centric (Zhang et al 2022); the human-centric evaluation sets consist of only 50 images from GAICD and 215 from FLMS and FCDB combined.…”

“…Since these datasets are not specific to a subject type (e.g., human), prior work on human-centric cropping (Zhang et al 2022) is limited by the number of images available for training and evaluation. Only 1.1K, 339, 176, and 39 of the images in CPC, GAICD, FCDB and FLMS (Fang et al 2014) are human-centric (Zhang et al 2022); the human-centric evaluation sets consist of only 50 images from GAICD and 215 from FLMS and FCDB combined. SACD (Yang et al 2023) is a recent dataset for subject-aware cropping that does not focus on a particular subject type but contains 24K+ labels and 5.2 million ranking pairs generated using their annotation procedure.…”

“…Framing and cropping decide what elements of a scene to include or exclude from the image, and how to frame or crop an image is influenced by the subject that one wishes to portray. Subjectaware cropping takes a notion of a subject in addition to pixels and has been studied in recent work on data-driven approaches (Yang et al 2023) and in the context of humancentric images (Zhang et al 2022). High-quality solutions to this problem are based on supervised learning, from large datasets of manual annotations created specifically for cropping (Zeng et al 2019;Wei et al 2018;Yang et al 2023).…”

“…To demonstrate the effectiveness of GenCrop, we evaluate it on the human-centric (portrait) and the subject-aware cropping tasks proposed by (Zhang et al 2022) and (Yang et al 2023). We show that GenCrop yields competitive results against fully supervised approaches (Zhang et al 2022) on the existing datasets (Fang et al 2014;Chen et al 2017a;Yang et al 2023) (under quantitative metrics such as Intersection-over-Union and boundary displacement (Zhang et al 2022)), while being superior to the best weakly/unsupervised method (Chen et al 2017b). We also evaluate Gen-Crop on additional subject categories such as cats, dogs, etc.…”