Concept-Based and Embedding-Based Models in Lifelog Retrieval: An Empirical Comparison of Performance

Abstract

Many lifelog retrieval systems have been introduced that apply various approaches to their search engines. The traditional method was to match concepts, which are visual objects detected in images and semantic queries. This concept-based approach has been applied in many retrieval systems, achieving the top performance in lifelog search challenges. Many novel embedding-based cross-modality retrieval models, such as CLIP, BLIP, or HADA, have been developed recently and obtained state-of-the-art (SOTA) results in the image-text retrieval task. These models have recently been applied in several lifelog search challenges. However, there is no comprehensive comparison between them since many benchmarking evaluations contain bias factors such as different user interfaces of participated lifelog retrieval systems. In this paper, we conducted non-biased experiments in both automatic (non-interactive) and interactive configurations to evaluate the performance of many SOTA retrieval models, including the traditional concept-based approach, in the lifelog retrieval task. Furthermore, we retrained the models in a lifelog Q&A dataset to assess whether retraining on a small lifelog dataset could improve the performance. The result showed that embedding-based search engines outperformed the concept-based approach by a large margin in both settings. The finding opens the opportunity to apply the embedding-based models as a new generation of lifelog retrieval models instead of the conventional concept-based approach. The source code will be available soon for reproducibility.