In the cultural heritage (CH) field, abstract concepts–such as comfort, power, or freedom–are considered powerful tools to index cultural visual data. Simultaneously, the ever-increasing number of images is too great to index manually, so computer vision (CV)-based methods are employed to automatically classify images. The CV field, however, has paid scarce attention to image classification based on abstract concepts, partly because it has historically favored tasks and datasets for purportedly objective, concrete categories. More subjective and culturally-coded classes, like abstract concepts, more explicitly require interpretability of predictions by automatic models, given the potential to echo harmful bias. This is especially true for their detection in CH collections, given that they tend to contain many `ethically sensitive' depictions. To address the gap in CV-based interpretable methods for automatic classification of abstract concepts, we (1) present ARTstract, a dataset of cultural images and their evocation of certain abstract concepts (2) report baseline model performances on ARTstract for the task of image classification by abstract concepts, and, critically, (3) use them as a case study of traditional and non-traditional approaches to visual interpretability, inspired by [Offert \& Bell 2021]’s work. We introduce a unique approach to testing the interpretability of predictions by automatic models, using a combination of attribution maps (AM) and stable diffusion (SD). This approach has not been attempted before, and we experiment with it as a way to extend hermeneutic work back into the technical system that enables, unveiling its potential as a provider of lessons for developing systems that are interpretable-by-design.