Batch-based activity recognition from egocentric photo-streams revisited

Abstract

Wearable cameras can gather large amounts of image data that provide rich visual information about the daily activities of the wearer. Motivated by the large number of health applications that could be enabled by the automatic recognition of daily activities, such as lifestyle characterization for habit improvement, context-aware personal assistance and tele-rehabilitation services, we propose a system to classify 21 daily activities from photo-streams acquired by a wearable photo-camera. Our approach combines the advantages of a Late Fusion Ensemble strategy relying on convolutional neural networks at image level with the ability of recurrent neural networks to account for the temporal evolution of high level features in photo-streams without relying on event boundaries. The proposed batch-based approach achieved an overall accuracy of 89.85%, outperforming state of the art end-to-end methodologies. These results were achieved on a dataset consists of 44,902 egocentric pictures from three persons captured during 26 days in average. arXiv:1710.04112v2 [cs.CV] 9 May 2018 1 The annotations are publicly available at https://www.github.com/gorayni/egocentric photostreams.• we extend significantly the comparisons with state of the art techniques.

Activities of Daily Living Monitoring via a Wearable Camera: Toward Real-World Applications

“…The life of three unrelated subjects was presented in the NTCIR-12 challenge [13]. Although it was independently annotated with 21 daily activity labels by [14], it only considers three people. Another dataset for image retrieval consisting of the annotated moments of two people [56] was released and further labeled in terms of four different activities.…”

“…In comparison with previous visual lifelog datasets, the activities are not constrained to a specific domain and occurred in a wide variety of indoor and outdoor locations of a city. The set of activity labels is based on previous works [14], [15], [59] and further expanded to 35 activities, thus adding 14 more categories. Moreover, the number of users is greater than in existing datasets by 12 people as seen in Table 1, allowing them to perform a generalization test.…”

“…The selected categories for the dataset are general activities from five different egocentric groups [1], as seen on Table 2. The activity labels were based on previous works [14], [15], [59], but they were not specifically targeted to model the student lifestyle and were selected after the recording. As an illustration, the original number of categories proposed for annotation included a broader set of activities such as child rearing, praying, painting or meditating.…”

“…This selection was made as proportional as possible with respect to the categories since it had to be representative of the dataset. In contrast with [14], we considered that this kind of partition is not enough to assess the generalization performance, because similar days might depict similar activities in the same context of a person. Consequently, we made another test split consisting of unseen users during training.…”

“…(iii) A set of experiments using the correlation alignment (CORAL) adaptation method [11], [12] showing that the model learned with our dataset can be easily and successfully transferred to other existing datasets acquired by two different wearable cameras (i.e. NTCIR-12 [13], [14], and Castro et al's [15] datasets), providing competitive results with a very little amount of labeled data.…”