Fully Automated Pose Estimation of Historical Images in the Context of 4D Geographic Information Systems Utilizing Machine Learning Methods

Abstract

The idea of virtual time machines in digital environments like hand-held virtual reality or four-dimensional (4D) geographic information systems requires an accurate positioning and orientation of urban historical images. The browsing of large repositories to retrieve historical images and their subsequent precise pose estimation is still a manual and time-consuming process in the field of Cultural Heritage. This contribution presents an end-to-end pipeline from finding relevant images with utilization of content-based image retrieval to photogrammetric pose estimation of large historical terrestrial image datasets. Image retrieval as well as pose estimation are challenging tasks and are subjects of current research. Thereby, research has a strong focus on contemporary images but the methods are not considered for a use on historical image material. The first part of the pipeline comprises the precise selection of many relevant historical images based on a few example images (so called query images) by using content-based image retrieval. Therefore, two different retrieval approaches based on convolutional neural networks (CNN) are tested, evaluated, and compared with conventional metadata search in repositories. Results show that image retrieval approaches outperform the metadata search and are a valuable strategy for finding images of interest. The second part of the pipeline uses techniques of photogrammetry to derive the camera position and orientation of the historical images identified by the image retrieval. Multiple feature matching methods are used on four different datasets, the scene is reconstructed in the Structure-from-Motion software COLMAP, and all experiments are evaluated on a newly generated historical benchmark dataset. A large number of oriented images, as well as low error measures for most of the datasets, show that the workflow can be successfully applied. Finally, the combination of a CNN-based image retrieval and the feature matching methods SuperGlue and DISK show very promising results to realize a fully automated workflow. Such an automated workflow of selection and pose estimation of historical terrestrial images enables the creation of large-scale 4D models.

Faroe Islands rephotography image registration dataset

“…The rephotographic compilations of this dataset are of value to several divisions of the scientific community: Computer Vision: The data consist of image pairs of historic monochrome images and contemporary colour images. Since the images are registered, they can be used to develop, train, and evaluate (deep) multitemporal image registration, especially those geared towards historical images [ 4 , 5 ]. Landscape Change: Image pairs depicting the Faroese landscape can be used to study and visualize the landscape and land usage change on the Faroe Islands.…”

“…Computer Vision: The data consist of image pairs of historic monochrome images and contemporary colour images. Since the images are registered, they can be used to develop, train, and evaluate (deep) multitemporal image registration, especially those geared towards historical images [ 4 , 5 ].…”

Icepy4d: A Python Toolkit for Advanced Multi-Epoch Glacier Monitoring With Deep-Learning Photogrammetry

“…Recently, DL-based feature matching algorithms have shown promising performance in challenging scenarios (Yao et al, 2021, Chen et al, 2021. For example, SuperGlue, an end-to-end Convolutional Neural Network (CNN) for feature extraction and matching, has proven effective in various real-world scenarios, including low-quality images from webcams (Wu et al, 2021), historical images (Maiwald et al, 2021) and images with different viewpoints or acquisition conditions (Bellavia et al, 2022). Another popular detector-free end-to-end matcher gaining traction is LOFTR (Sun et al, 2021), which is also finding applications in the computer vision and remote sensing communities (Bellavia et al, 2022, Liang et al, 2023.…”

Photogrammetry Now and Then – From Hand-Crafted to Deep-Learning Tie Points –

“…Furthermore, an alternative evaluation metric, previously proposed in (Bellavia, 2022d), that exploits rough optical flow estimation and the epipolar error, is employed. Although the reported evaluations employ only image pairs, the obtained results are useful also in the case of one-to-many registration of multi-temporal images, as those employed in AR/VR applications and in multi-view historical photogrammetric applications (Maiwald et al, 2021), since Structure-from-Motion (SfM) relies on the single image pair matching as base step.…”

“…Better and more robust DNNs are progressively appearing to improve scale, rotation, and radiometric invariances as well as the localization accuracy, the repeatability and the reliability of the keypoints. Several of these image matching DNNs have already been successfully applied and tested in satellite and aerial multitemporal datasets (Ghuffar et al, 2022;Zhang et al, 2021;Farella et al, 2022) and terrestrial multi-temporal historical image pairs (Maiwald et al, 2021).…”