Everyday concept detection in visual lifelogs: validation, relationships and trends

Abstract

The Microsoft SenseCam is a small lightweight wearable camera used to passively capture photos and other sensor readings from a user's day-to-day activities. It captures on average 3,000 images in a typical day, equating to almost 1 million images per year. It can be used to aid memory by creating a personal multimedia lifelog, or visual recording of the wearer's life. However the sheer volume of image data captured within a visual lifelog creates a number of challenges, particularly for locating relevant content. Within this work, we explore the applicability of semantic concept detection, a method often used within video retrieval, on the domain of visual lifelogs. Our concept detector models the correspondence between low-level visual features and high-level semantic concepts (such as indoors, outdoors, people, buildings, etc.) using supervised machine learning. By doing so it determines the probability of a concept's presence. We apply detection of 27 everyday semantic concepts on a lifelog collection composed of 257,518 SenseCam images from 5 users. The results were evaluated on a subset of 95,907 images, to determine the accuracy for detection of each semantic concept. We conducted further analysis on the temporal consistency, co-occurance and relationships

Beyond Shot Retrieval: Searching for Broadcast News Items Using Language Models of Concepts

“…This probability can for example be estimated by a method described by Platt [14], which considers each concept occurrence independently. Work in the somewhat related domain of lifelogging has shown that the occurrences of many concepts within a shot and within adjacent shots is statistically dependent on each other [3]. However in this work we concentrate on more generic representations and leave the investigations of these dependencies for future work.…”

“…Given the growing prominence and attention afforded to lifelog data from wearable cameras such as the SenseCam, where audio data isn't recorded [3], we want our model to be also applicable to search in video data without considering the audio stream. As a result we focus on working with concepts extracted from the content of images.…”

Unconscious emotions: quantifying and logging something we are not aware of

“…Some people tend to write down in their diaries all the details of what they saw and did, while others like to note moods and emotions they had during a day. Presently, there are various kinds of lifelogging tools (e.g., [1][2][3]) that have been developed to assist people with recording their life experiences. However, these tools can only record the surrounding environment of people, which ultimately includes everything that they encounter, but not the internal world, which comprises moods, thoughts and emotions.…”

Semantics-based selection of everyday concepts in visual lifelogging

“…In state-of-the-art everyday concept detection and validation [8], concepts are suggested by several SenseCam users after they have gone through and studied several days' of their own lifelogged events. Then, being more familiar with their own lifestyles through reviewing their own lifelogs, concepts are discussed and filtered with the added criterion that the concept can be detected with satisfactory accuracy.…”