Activities of Daily Living Monitoring via a Wearable Camera: Toward Real-World Applications

Abstract

Activity recognition from wearable photo-cameras is crucial for lifestyle characterization and health monitoring. However, to enable its wide-spreading use in real-world applications, a high level of generalization needs to be ensured on unseen users. Currently, state-of-the-art methods have been tested only on relatively small datasets consisting of data collected by a few users that are partially seen during training. In this paper, we built a new egocentric dataset acquired by 15 people through a wearable photo-camera and used it to test the generalization capabilities of several state-of-the-art methods for egocentric activity recognition on unseen users and daily image sequences. In addition, we propose several variants to state-of-the-art deep learning architectures, and we show that it is possible to achieve 79.87% accuracy on users unseen during training. Furthermore, to show that the proposed dataset and approach can be useful in real-world applications, where data can be acquired by different wearable cameras and labeled data are scarcely available, we employed a domain adaptation strategy on two egocentric activity recognition benchmark datasets. These experiments show that the model learned with our dataset, can easily be transferred to other domains with a very small amount of labeled data. Taken together, those results show that activity recognition from wearable photo-cameras is mature enough to be tested in real-world applications. INDEX TERMS Daily activity recognition, visual lifelogs, domain adaptation, wearable cameras.

Understanding Event Boundaries for Egocentric Activity Recognition from Photo-Streams

“…We carried out our experiments on the ADLEgoDataset [6], a visual lifelogging dataset collected using the Narrative Camera. This dataset consists of 125 egocentric photostreams with 35 activity categories recorded by 15 students on their daily routine.…”

“…These testing splits contain full-day sequences not present in the training split, and their data percentage for the unseen and seen users was around 5% and 10%, respectively. In contrast to [6], we discarded the categories that were only performed by one participant, as the model would probably overfit that category. Moreover, we also removed the categories that had less than 200 instances, since we considered that they had a few instances for training a convolutional model.…”

“…In the case of action recognition, we considered that using only the accuracy for measuring the model performance would be misleading as the testing splits are highly imbalanced. Therefore, we also used mean average precision (mAP) and other macro metrics for precision and recall as in [6]. In the case of action detection, we measured the mAP as the overlap of intersection over the union (IoU) with different thresholds as defined in [33].…”

“…In this work, we take a step forward in this direction by investigating the questions: How important are event boundaries for activity recognition from egocentric photostreams?, and Does the temporal coherence of segmented events from egocentric photostreams improves the activity recognition performance at the frame-level?. In [6], it was demonstrated that the training strategy directly affects the performance of the activity classifier. The core contribution of this work is to evaluate a training strategy for activity recognition based on temporal boundaries that define events.…”

Multiple Human Tracking in Non-Specific Coverage with Wearable Cameras

“…Most existing methods use fixed cameras for video collection, whose field of view (FOV) is unchanged and limited. In contrast, wearable cameras, e.g., GoPro and Google glass, worn by and moved with wearers, have time-varying non-specific observation coverage [1][2][3][4][5] and can be used to track and observe people at different sites by varying the camera views, which enables more flexible and wide-range outdoor video surveillance of crowded scenes. The goal of this paper is to study the new problem of MHT in non-specific fields using wearable cameras.…”