Using visual lifelogs to automatically characterize everyday activities

Abstract

Visual lifelogging is the term used to describe recording our everyday lives using wearable cameras, for applications which are personal to us and do not involve sharing our recorded data. Current applications of visual lifelogging are built around remembrance or searching for specific events from the past. The purpose of the work reported here is to extend this to allow us to characterise and measure the occurrence of everyday activities of the wearer and in so doing to gain insights into the wearer's everyday behaviour.The methods we use are to capture everyday activities using a wearable camera called SenseCam, and to use an algorithm we have developed which indexes lifelog images by the occurrence of basic semantic concepts. We then use data reduction techniques to automatically generate a profile of the wearer's everyday behaviour and activities. Our algorithm has been evaluated on a large set of concepts investigated from 13 users in a user experiment, and for a group of 16 popular everyday activities we achieve an * Corresponding author at: CLARITY: Centre for Sensor Web Technologies and School of Computing, Dublin City University, Glasnevin, Dublin 9, Ireland. Tel: +353 -1 -7005262Email address: alan.smeaton@dcu.ie (Alan F. Smeaton) January 12, 2013 average F-score of 0.90. Our conclusions are that the the technique we have presented for unobtrusively and ambiently characterising everyday behaviour and activities across individuals is of sufficient accuracy to be usable in a range of applications. Preprint submitted to Information Sciences

Computer Vision for Lifelogging

“…Another good example of this is the work by Browne et al [23] who used the visual lifelog from a SenseCam to stimulate autobiographical recollection, promoting consolidation and retrieval of memories for significant events. All these clinical explorations seem to agree that visual lifelogging provides a "powerful boost to autobiographical recall, with secondary benefits for quality of life" [23,3].…”

“…Compared to traditional digital monitoring through which users are monitored by others, lifelogging introduces a new form of sousveillance, i.e., capturing data about oneself for use by oneself [1,2]. This is opposed to having somebody else record what we are doing and using the logged data for some public or shared purpose [3]. Among various applications of lifelogging, assistive living accounts for an important part of lifelogging research aiming at improving the health and well-being of human both mentally (such as memory recall) and physically (such as anomaly detection), thanks to the advantages of lifelogging in measuring activities longitudinally in fine granularity.…”

“…Because visual information contains more semantics of events which can be used to infer other contextual information like 'who, what, where and when', visual lifelogging can usually act as the prosthesis of the lifeloger's experience and this forms a stream of lifelogging research and assistive living. As the enabler for visual lifelogging, camera-enabled sensors are used in wearable devices to record still images [40] or video [3,13,32] taken from a first-person view, i.e. representing the subject's view of everyday activities.…”

“…Visual lifelogging has already been widely applied in assistive living applications including aiding human memory recall, diet monitoring, chronic disease diagnosis, recording activities of daily living and so on. Example visual lifelogging projects include Steve Manns WearCam [31,32] , the DietSense project at UCLA [38] , the WayMarkr project at New York University [4] , the InSense system at MIT [3] , and the IMMED system [33] . Microsoft Research catalysed research in this area with the development of the SenseCam [12,40] which was made available to other research groups in the late 2000s.…”

“…This is also tested in a more comprehensive visual lifelogging experiment in [56]. Similar work is also carried out using concept detections to characterize everyday activities as reported in [57,3] where activity recognition is built on the basis of concept detection. In this chapter, we investigate a selection of attributebased recognition methods which are suitable for activity characterization in assistive living.…”