Connecting Images through Sources: Exploring Low-Data, Heterogeneous Instance Retrieval

Abstract

Along with a new volume of images containing valuable information about our past, the digitization of historical territorial imagery has brought the challenge of understanding and interconnecting collections with unique or rare representation characteristics, and sparse metadata. Content-based image retrieval offers a promising solution in this context, by building links in the data without relying on human supervision. However, while the latest propositions in deep learning have shown impressive results in applications linked to feature learning, they often rely on the hypothesis that there exists a training dataset matching the use case. Increasing generalization and robustness to variations remains an open challenge, poorly understood in the context of real-world applications. Introducing the alegoria benchmark, containing multi-date vertical and oblique aerial digitized photography mixed with more modern street-level pictures, we formulate the problem of low-data, heterogeneous image retrieval, and propose associated evaluation setups and measures. We propose a review of ideas and methods to tackle this problem, extensively compare state-of-the-art descriptors and propose a new multi-descriptor diffusion method to exploit their comparative strengths. Our experiments highlight the benefits of combining descriptors and the compromise between absolute and cross-domain performance.

Giving Historical Photographs a New Perspective: Introducing Camera Orientation Parameters as New Metadata in a Large-Scale 4D Application

“…This includes, among others, maps, building models, images, and texts in combined virtual environments [16]. Most tools rely on web-based interfaces that combine different sources [17,18], enabling advanced searching [19] and user management [20]. Others develop comprehensive frameworks for creating Web3D Cultural Heritage applications [21].…”

ALEGORIA: Joint Multimodal Search and Spatial Navigation into the Geographic Iconographic Heritage

“…The content-based part of the indexing had to deal with image descriptors tuned to the contents considered. We have studied generalizable and robust deep descriptors for the problem of low-data, heterogeneous image retrieval, as well as post-processings to improve retrieval, such as geometric verification and query expansion [5]. These modalities are made available to query one or several collections together, by combining metadata and image content search (through the query-by-example paradigm and various image descriptors).…”

Towards Culture-Aware Smart and Sustainable Cities: Integrating Historical Sources in Spatial Information Infrastructures

“…Content-based retrieval methods were developed during the French projects ALEGORIA and Archival City. ALEGORIA develops multimodal and content-based indexing and visualization tools, to facilitate the promotion of iconographic institutional funds collections, describing the French territory in various periods, from the interwar period to our days [8]. Archival City relies on smart and dedicated tools for accessing, viewing, and using the city archives of the following specific places: Greater Paris, Algiers, Bologna, Chiang-Mai, Jerusalem, and Quito [9].…”