Interactive Multimodal Lifelog Retrieval with vitrivr at LSC 2021

Abstract

The Lifelog Search Challenge (LSC) is an annual benchmarking competition for interactive multimedia retrieval systems, where participating systems compete in finding events based on textual descriptions containing hints about structured, semi-structured, and/or unstructured data. In this paper, we present the multimedia retrieval system vitrivr, a long-time participant to LSC, with a focus on new functionality. Specifically, we introduce the image stabilisation module which is added prior to the feature extraction to reduce the image degradation caused by lifelogger movements, and discuss how geodata is used during query formulation, query execution, and result presentation. CCS CONCEPTS• Information systems → Search interfaces; Image search; Users and interactive retrieval; Multimedia databases; Information retrieval; • Human-centered computing → Interactive systems and tools.

MEMORIA: A Memory Enhancement and MOment RetrIeval Application for LSC 2022

“…Similarly, Voxento 2.0 [2], provides a voice interface for interactive search engine and uses a speech to text method to generate queries. The vitrivr retrieval system [18] supports several multimedia formats and query modalities.…”

vitrivr at the Lifelog Search Challenge 2022

“…vitrivr is a "a scalable, content-based multimedia information retrieval stack" [3]. Its general purpose nature allows usage in video retrieval [9,35], cultural heritage [22,30], and it is a regular participant at LSC [11,14,23] with its core architecture unchanged in the last few iterations. The system has three core components: a database specialized for multimedia retrieval, Cottontail DB [2], a retrieval engine, Cineast [27], and the web-based user interface, vitrivr-ng.…”

“…Frontend: vitrivr-ng is "responsible for query formulation, result presentation, browsing and filtering. Users can combine various query modalities, for example, first filtering by time and then making a color sketch" [11]. Figures 2 and 3 show the current UI.…”

“…In this paper, we present the vitrivr 1 multimedia retrieval system [28], which has participated in previous iterations of LSC [11,14,23]. We introduce vitrivr's system architecture and retrieval model and demonstrate how lifelog queries are solved with vitrivr using example queries.…”

Memento 2.0: An Improved Lifelog Search Engine for LSC'22

“…LSC '22, June 27-30, 2022, Newark, NJ, USA vitrivr-VR is an extension of the system vitrivr [14] and uses the same backend comprised of Cineast which is a feature extraction and query processing engine along with CottontailDB [8] while ViRMA used the 𝑀 3 model which translates the data into multidimensional space and is explored via the VR interface by projecting it into 3-D space. Similar to ViRMA [7], PhotoCube [22] also used the 𝑀 3 model and a web interface for search and navigation by creating a 3-D exploration cube on the screen.…”