Machine Learning Based Analysis of Finnish World War II Photographers

Abstract

In this paper, we demonstrate the benefits of using state-of-the-art machine learning methods in the analysis of historical photo archives. Specifically, we analyze prominent Finnish World War II photographers, who have captured high numbers of photographs in the publicly available Finnish Wartime Photograph Archive, which contains 160,000 photographs from Finnish Winter, Continuation, and Lapland Wars captures in 1939-1945. We were able to find some special characteristics for different photographers in terms of their typical photo content and framing (e.g., close-ups vs. overall shots, number of people). Furthermore, we managed to train a neural network that can successfully recognize the photographer from some of the photos, which shows that such photos are indeed characteristic for certain photographers. We further analyzed the similarities and differences between the photographers using the features extracted from the photographer classifier network. We make our annotations and analysis pipeline publicly available, in an effort to introduce this new research problem to the machine learning and computer vision communities and facilitate future research in historical and societal studies over the photo archives.

Automatic Image Content Extraction: Operationalizing Machine Learning in Humanistic Photographic Studies of Large Visual Archives

“…Especially for the historical photos, the camera technical variables may be also a study question as such to analyze the development of photography. Predicting the photographer from a general photo is naturally almost impossible, but may be possible to some extent when the possible pool of photographers is limited (Chumachenko et al 2020). The variables in '2 Composition' and '3 Modality' correspond to the "visual form" that is mostly easy to evaluate from the images.…”

“…The description of the specific methods available for different tasks is beyond the scope of this paper, but interested readers may refer to articles in computer vision and machine learning fields to look for more information. A list of state-of-the-art methodologies for the different problems is: classification (Krizhevsky, Sutskever, and G. E. Hinton 2012), photographer recognition and framing classification (Chumachenko et al 2020), camera pose estimation (Kendall and Cipolla 2017), basic color analysis (Gonzalez and Woods 2018), salience estimation (G. Li and Y. Yu 2015), main character recognition (Seker et al 2021a), age and gender estimation (Rodriguez et al 2017), face recognition (M. Wang and Deng 2021), height and weight estimation (Altinigne, Thanou, and Achanta 2020), nudity detection (Ion and Minea 2019), clothes recognition (Z. Liu et al 2016), clothing style clustering (Matzen, Bala, and Snavely 2017), person detection (Braun et al 2019), gaze estimation (Kellnhofer et al 2019), social distance estimation (Seker et al 2021b), optical character recognition (Memon et al 2020), object detection (L. Liu et al 2020), scene recognition (Zhou et al 2017), event recognition (L. Wang et al 2018), weather recognition (Zhao et al 2018), scene segmentation (Fu et al 2019), expression recognition (S. Li and Deng 2020), pose estimation (Cao et al 2019), action recognition (Dong et al 2021), visual relationship detection (R. Yu et al 2017), human interaction recognition (Stergiou and Poppe 2019), human-object interaction detection (Y.-L. Li et al 2019), content-based image retrieval (Tzelepi and Tefas 2018), and clustering (Min et al 2018).…”

“…This can allow revealing such similarities and groupings of data that the researcher would not be able to look for. There are also tools for visualizing such found similarities and groupings as demonstrated in a recent paper (Chumachenko et al 2020), where the similarities between different photographers were illustrated based on clustering their photos. A simplified illustration of clustering images into groups based on the similarity of poses is provided in Fig.…”

“…) demonstrated the use of a pretrained CNN model for three different automatic visual content analysis tasks over a collection of 450,000 historical images: 1) determining whether an image is an illustration or photograph and defining the historical watershed moment when the number of photographs overtook the number of illustrations, 2) querying images based on abstract visual aspects (clustering visually similar advertisements), and 3) retraining the last layer of the network to classify images into nine relevant visual categories defined by domain experts. A recent study by Chumachenko et al (2020), co-authored by most of the authors of this papers, demonstrated the use of CNNs for conducting a novel kind of analysis of prominent Finnish World War II photographers. A CNN was trained to recognize the photographer from the photo, which allowed learning some abstract similarities and differences between the photographers.…”

Ensembling Object Detectors for Image and Video Data Analysis

“…Driven by the broad availability of an extensive amount of datasets in different domains, object detection has become one of the most widely used tools within the field of computer vision in recent years, finding applications in various areas, such as video surveillance [1], medical diagnostics [2], historical image analysis [3], and industrial applications [4]. Despite the huge progress made in the field of object detection in the recent years, not much attention has been paid to the generalization ability of the object detection methods: the developed algorithms assume that the training and test data come from the same distribution, and the test performance is reported on data from the same dataset used for training.…”