A-Lamp: Adaptive Layout-Aware Multi-patch Deep Convolutional Neural Network for Photo Aesthetic Assessment

Abstract

Deep convolutional neural networks (CNN) have recently been shown to generate promising results for aesthetics assessment. However, the performance of these deep CNN methods is often compromised by the constraint that the neural network only takes the fixed-size input. To accommodate this requirement, input images need to be transformed via cropping, warping, or padding, which often alter image composition, reduce image resolution, or cause image distortion. Thus the aesthetics of the original images is impaired because of potential loss of fine grained details and holistic image layout. However, such fine grained details and holistic image layout is critical for evaluating an image's aesthetics. In this paper, we present an Adaptive Layout-Aware Multi-Patch Convolutional Neural Network (A-Lamp CNN) architecture for photo aesthetic assessment. This novel scheme is able to accept arbitrary sized images, and learn from both fined grained details and holistic image layout simultaneously. To enable training on these hybrid inputs, we extend the method by developing a dedicated double-subnet neural network structure, i.e. a Multi-Patch subnet and a Layout-Aware subnet. We further construct an aggregation layer to effectively combine the hybrid features from these two subnets. Extensive experiments on the largescale aesthetics assessment benchmark (AVA) demonstrate significant performance improvement over the state-of-theart in photo aesthetic assessment.

Effective Aesthetics Prediction With Multi-Level Spatially Pooled Features

“…Some existing works such as [4,14] use additional information about each image. We only compare methods that use purely information derived from image content.…”

“…For instance, [4] use user comments to enrich the available information for each image. While this latter source is clearly incompatible with image content, Ma et al [14] indirectly use information about the actual resolution of each image by including it in attribute graphs. Thus [14] reports a higher accuracy of 82.5% when using extra information, while their performance based on image content alone is 81.70%.…”

“…Both networks employ custom shallow shared-weights CNNs for each column. Ma et al [14] further extend Lu et al's work [12] by selecting 224 × 224 patches non-randomly based on image saliency and aggregating them while modeling their relative layout. Instead of taking patches, Mai et al [15] use a multi-column architecture to integrate five spatial pooling sizes, which allows for testing with variable resolution images.…”

“…An important use case for an aesthetics assessment model is its ability to establish ranking relationships between the quality of images. This can help Model SRCC PLCC Accuracy Murray et al [17] --66.70% Kao et al [5] --71.42% Lu et al [12] --74.46% Lu et al [11] --75.42% Hii et al [4] 75.76% Mai et al [15] --77.40% Kong et al [9] 0.558 -77.33% Talebi et al [22] V2 0.612 0.636 81.51% Talebi et al [22] V3 0.639* 0.645* 72.30%* Ma et al [14] --81.70%…”

Aesthetic Attributes Assessment of Images

“…Before deep learning era, many hand-crafted features [7,16] are designed for aesthetic image classification and scoring as surveyed by Deng et al [9]. Deep learning methods are proposed recently for aesthetic assessment [11,14,15,17,18,20,21,23,24,31]. They outperform traditional methods.…”