Semantics-based selection of everyday concepts in visual lifelogging

Abstract

Concept-based indexing, based on identifying various semantic concepts appearing in multimedia, is an attractive option for multimedia retrieval and much research tries to bridge the semantic gap between the media's low-level features and high-level semantics. Research into concept-based multimedia retrieval has generally focussed on detecting concepts from high quality media such as broadcast TV or movies, but it is not well addressed in other domains like lifelogging where the original data is captured with poorer quality. We argue that in noisy domains such as lifelogging, the management of data needs to include semantic reasoning in order to deduce a set of concepts to represent lifelog content for applications like searching, browsing or summarisation. Using semantic concepts to manage lifelog data relies on the fusion of automaticallydetected concepts to provide a better understanding of the lifelog data. In this paper, we investigate the selection of semantic concepts for lifelogging which includes reasoning on semantic networks using a density-based approach. In a series of experiments we compare different semantic reasoning approaches and the experi-

Computer Vision for Lifelogging

“…As introduced in Section 3.2, appropriate selection of semantic concepts as attributes impacts the recognition accuracy of activities. Though manual construction of semantic space such as topic-related concept selection user experiment [92,93], has shown its merits, such method is less flexible when dealing with various activity types which are not existing in the pre-defined activity set. A more feasible solution is to exploit external online resources in order to reflect human knowledge on activity-specific concept selection.…”

What are the Limits to Time Series Based Recognition of Semantic Concepts?

“…For lifelog activity recognition, the set of 85 everyday concepts investigated in [12] are used as semantic attributes. Dataset1 includes event samples of 16 activity types collected from 4 SenseCam wearers and consisting of 10,497 SenseCam images [11].…”

Semantically Smoothed Refinement for Everyday Concept Indexing

“…Although individual wearers may have different contexts and personal characteristics, there is a common understanding of concepts that is already socially agreed and allows people to communicate about these according to [6] and [17]. This makes it reliable for users to choose suitable concepts relevant to activities.…”

“…The aim of the user experiment was to determine candidate semantic concepts which have high correlation with various human activities. After several iterations and refinements we selected 85 base concepts [17,15] which have highest agreement among participants, i.e. more than half of respondents think each are relevant to the activity.…”

“…To assess the performance of our algorithm, we used a set of 85 everyday concepts [17,15] introduced in Section 3. We used a dataset including event samples of 23 activity types collected from 4 SenseCam wearers and consisting of 12,248 SenseCam images [15].…”