A 4D information system for the exploration of multitemporal images and maps using photogrammetry, web technologies and VR/AR

Abstract

<p class="VARAbstract">This contribution shows the comparison, investigation, and implementation of different access strategies on multimodal data. The first part of the research is structured as a theoretical part opposing and explaining the terms of conventional access, virtual archival access, and virtual museums while additionally referencing related work. Especially, issues that still persist in repositories like the ambiguity or missing of metadata is pointed out. The second part explains the practical implementation of a workflow from a large image repository to various four-dimensional applications. Mainly, the filtering of images and in the following, the orientation of images is explained. Selection of the relevant images is partly done manually but also with the use of deep convolutional neural networks for image classification. In the following, photogrammetric methods are used for finding the relative orientation between image pairs in a projective frame. For this purpose, an adapted Structure from Motion (SfM) workflow is presented, in which the step of feature detection and matching is replaced by the Radiant-Invariant Feature Transform (RIFT) and Matching On Demand with View Synthesis (MODS). Both methods have been evaluated on a benchmark dataset and performed superior than other approaches. Subsequently, the oriented images are placed interactively and in the future automatically in a 4D browser application showing images, maps, and building models Further usage scenarios are presented in several Virtual Reality (VR) and Augmented Reality (AR) applications. The new representation of the archival data enables spatial and temporal browsing of repositories allowing the research of innovative perspectives and the uncovering of historical details.</p><p>Highlights:</p><ul><li>Strategies for a completely automated workflow from image repositories to four-dimensional (4D) access approaches.</li><li>The orientation of historical images using adapted and evaluated feature matching methods.</li><li>4D access methods for historical images and 3D models using web technologies and Virtual Reality (VR)/Augmented Reality (AR).</li></ul><p> </p>

Interactive virtual representation of the disappeared convent of El Carmen (Logroño) generated from a paper craft model

“…The importance of the urban context for the correct understanding of the building has been stressed. In this particular case, we have worked with a single element (the convent) located on a city plan from a particular moment in time and just for generating images for the exhibition, but this approach can be extended to more buildings and landscape, different epochs and for multiple purposes (Kersten et al, 2012;Spanò, Chiabrando, Dezzani, & Prencipe, 2016;Maiwald, Bruschke, Lehmann, & Niebling, 2019;Walmsley & Kersten, 2019).…”

An automatic workflow for orientation of historical images with large radiometric and geometric differences

“…Nonetheless, for the presented datasets (see the Datasets section), it has been noticed that SIFT and its variants RootSIFT (Arandjelović and Zisserman, 2012) and DSP‐SIFT (Dong and Soatto, 2015) did not perform well and produced only few, or even no, matching image pairs. Further algorithmic methods that have been evaluated on historical images (Maiwald et al, 2019; Maiwald, 2019) are maximally stable extremal regions (MSER: Matas et al, 2004), radiation‐invariant feature transform (RIFT: J. Li et al, 2020), matching on demand with view synthesis (MODS: Mishkin et al, 2015a) and MODS‐WxBS (Mishkin et al, 2015b). Initially, the MSER method developed for wide baselines between image pairs produced decent results for historical image pairs, but was outperformed by a combination of RIFT and MODS/MODS‐WxBS.…”

“…For SIFT and RIFT the already determined results (Maiwald et al, 2019; Maiwald, 2019) are used for a comparison. For MODS both published variations are used (MODS and MODS‐WxBS) to generate matches between the image pairs.…”

Geospatial platforms and immersive tools for social cohesion: the 4D narrative of architecture of Australia’s Afghan cameleers

“…The site is included in Europeana's 3D-ICONS (3D Digitisation of Icons of European Architectural and Archaeological Heritage) interactive geographic information platform. 2 Nowadays, advanced VR/AR, web technologies and photogrammetry enable users to interact with 3D buildings through time using different means such as multi-temporal images (Maiwald, Bruschke, Lehmann, & Niebling, 2019).…”