In this paper, we introduce a new lifelog retrieval system called Memento that leverages semantic representations of images and textual queries projected into a common latent space to facilitate effective retrieval. It bridges the semantic gap between complex visual scenes/events and user information needs expressed as textual and faceted queries. The system, developed for the 2021 Lifelog Search Challenge, also has a minimalist user interface that includes primary search, temporal search, and visual data filtering components. CCS CONCEPTS• Information systems → Information retrieval; Search interfaces; Retrieval models and ranking.

In this paper, we present Memento 2.0, an improved version of our system which first participated in the Lifelog Search Challenge 2021. Memento 2.0 employs image-text embeddings derived from two CLIP models (ViT-L/14 and ResNet-50x64) and adopts a weighted ensemble approach to derive a combined final ranking. Our approach significantly improves the performance over the baseline LSC'21 system. We additionally make important updates to the system's user interface after analysing the shortcomings to make it more efficient and better suited to the needs of the Lifelog Search Challenge. CCS CONCEPTS• Information systems → Retrieval models and ranking; Search interfaces.

The Lifelog Search Challenge (LSC) is an interactive benchmarking evaluation workshop for lifelog retrieval systems. The challenge was first organised in 2018 aiming to find the system that can quickly retrieve relevant lifelog images for a given semantic query. This paper provides an analysis of the performance of all 17 systems participating in the 4th LSC workshop held at the 2021 Annual ACM International Conference on Multimedia Retrieval (ICMR). LSC'21 was the largest effort at comparing different approaches to interactive lifelog retrieval systems seen thus far. Findings from the challenge suggest that many different interactive factors contribute to the success (or otherwise) of participating teams. In this paper, we provide an overview of the LSC'21 challenge, introduce each team's approach and explore these factors in depth and offer clues on how to develop a high-performing interactive lifelog search engine. INDEX TERMS lifelog, information retrieval, multimodal, analyticsAt LSC'21, each of the participating teams brought a unique and customised search engine to the challenge. In this paper, we introduce the LSC challenge, describe all competing systems, and highlight the techniques and components that are employed in state-of-the-art interactive lifelog

No abstract

In this extended paper, we describe our lifelog retrieval system called Memento which participated in the 2021 Lifelog Search Challenge in detail. Memento leverages semantic representations of images and textual queries projected into a common latent space to facilitate effective retrieval, aiming to bridge the existing semantic gap between complex visual scenes/events and user information needs expressed as textual and faceted queries. Our system also has a minimalist user interface which includes functionalities such as visual data filtering and temporal search. Finally, we include a comparative analysis of Memento’s performance at LSC 2021 and suggest improvements for future iterations of the system.