Multimodal Retrieval through Relations between Subjects and Objects in Lifelog Images

Chu, Tai-Te; Chang, Chia-Chun; Yen, An-Zi; Huang, Hen-Hsen; Chen, Hsin-Hsi

doi:10.1145/3379172.3391723

Cited by 15 publications

(13 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…FIRST [25] uses an autoencoder like approach to map query text and images into a common semantic space to measure the similarity between them, LifeGraph [23] used a knowledge graph to represent the lifelog data to capture the internal relations of the various data modalities and linked it to external static data sources for better semantic understanding. Chu et al [6] extracted relation graphs from lifelog images to better describe the relationship between entities (subject-object) present within the image.…”

Section: Related Workmentioning

confidence: 99%

Memento: A Prototype Lifelog Search Engine for LSC'21

Alam

Graham

Gurrin

2021

Proceedings of the 4th Annual on Lifelog Search Challenge

View full text Add to dashboard Cite

In this paper, we introduce a new lifelog retrieval system called Memento that leverages semantic representations of images and textual queries projected into a common latent space to facilitate effective retrieval. It bridges the semantic gap between complex visual scenes/events and user information needs expressed as textual and faceted queries. The system, developed for the 2021 Lifelog Search Challenge, also has a minimalist user interface that includes primary search, temporal search, and visual data filtering components. CCS CONCEPTS• Information systems → Information retrieval; Search interfaces; Retrieval models and ranking.

show abstract

Section: Related Workmentioning

confidence: 99%

Memento: A Prototype Lifelog Search Engine for LSC'21

Alam

Graham

Gurrin

2021

Proceedings of the 4th Annual on Lifelog Search Challenge

View full text Add to dashboard Cite

show abstract

“…Most of the systems offer querying by location. For our work the systems Myscéal [29], LifeSeeker [14], and the work of Chu et al [3] are particularly interesting, since they provide a visualization of spatial context with the help of a map in various different ways:…”

Section: Related Workmentioning

confidence: 99%

Interactive Multimodal Lifelog Retrieval with vitrivr at LSC 2021

Heller

Gasser

Parian-Scherb

et al. 2021

Proceedings of the 4th Annual on Lifelog Search Challenge

View full text Add to dashboard Cite

The Lifelog Search Challenge (LSC) is an annual benchmarking competition for interactive multimedia retrieval systems, where participating systems compete in finding events based on textual descriptions containing hints about structured, semi-structured, and/or unstructured data. In this paper, we present the multimedia retrieval system vitrivr, a long-time participant to LSC, with a focus on new functionality. Specifically, we introduce the image stabilisation module which is added prior to the feature extraction to reduce the image degradation caused by lifelogger movements, and discuss how geodata is used during query formulation, query execution, and result presentation. CCS CONCEPTS• Information systems → Search interfaces; Image search; Users and interactive retrieval; Multimedia databases; Information retrieval; • Human-centered computing → Interactive systems and tools.

show abstract

“…For this experiment, we used the LSC'18/19 dataset [14], and we created a new set of twenty semantic queries, including ten randomly chosen topics from the LSC'19 dataset (representing conventional lifelog queries) and ten manually created topics that focus on visually describing a known-item from a lifelog. It is noticeable that [6] also followed the concept of using a scene graph for lifelogging visual data. However, this system used such a graph as a supplement to the retrieval process and did not consider a query as a graph like our proposed method.…”

Section: Related Workmentioning

confidence: 99%

Graph-Based Indexing and Retrieval of Lifelog Data

Nguyen

Gurrin

2021

MultiMedia Modeling

View full text Add to dashboard Cite

Understanding the relationship between objects in an image is an important challenge because it can help to describe actions in the image. In this paper, a graphical data structure, named "Scene Graph", is utilized to represent an encoded informative visual relationship graph for an image, which we suggest has a wide range of potential applications. This scene graph is applied and tested in the popular domain of lifelogs, and specifically in the challenge of known-item retrieval from lifelogs. In this work, every lifelog image is represented by a scene graph, and at retrieval time, this scene graph is compared with the semantic graph, parsed from a textual query. The result is combined with location or date information to determine the matching items. The experiment shows that this technique can outperform a conventional method.

show abstract

Multimodal Retrieval through Relations between Subjects and Objects in Lifelog Images

Cited by 15 publications

References 4 publications

Memento: A Prototype Lifelog Search Engine for LSC'21

Memento: A Prototype Lifelog Search Engine for LSC'21

Interactive Multimodal Lifelog Retrieval with vitrivr at LSC 2021

Graph-Based Indexing and Retrieval of Lifelog Data

Contact Info

Product

Resources

About