Toyota Smarthome: Real-World Activities of Daily Living

Abstract

The performance of deep neural networks is strongly influenced by the quantity and quality of annotated data. Most of the large activity recognition datasets consist of data sourced from the web, which does not reflect challenges that exist in activities of daily living. In this paper, we introduce a large real-world video dataset for activities of daily living: Toyota Smarthome. The dataset consists of 16K RGB+D clips of 31 activity classes, performed by seniors in a smarthome. Unlike previous datasets, videos were fully unscripted. As a result, the dataset poses several challenges: high intra-class variation, high class imbalance, simple and composite activities, and activities with similar motion and variable duration. Activities were annotated with both coarse and fine-grained labels. These characteristics differentiate Toyota Smarthome from other datasets for activity recognition. As recent activity recognition approaches fail to address the challenges posed by Toyota Smarthome, we present a novel activity recognition method with attention mechanism. We propose a pose driven spatiotemporal attention mechanism through 3D ConvNets. We show that our novel method outperforms state-of-the-art methods on benchmark datasets, as well as on the Toyota Smarthome dataset. We release the dataset for research use 1 .

Improved Action Recognition with Separable Spatio-Temporal Attention Using Alternative Skeletal and Video Pre-Processing

“…For this reason, Das et al [ 17 ] presented a dataset for action recognition with the particularities of ADLs: namely, one that has lower inter-class variation than usual in other general action recognition datasets, while having greater intra-class variation by different users. Their dataset consists of 16,115 videos, spanning 31 activity classes taken from 7 different views (not necessarily concurrently, though).…”

“…There are two main limitations in the STA solution proposed by Das et al [ 17 ]. The first one has to do with how skeletons are fed into the LSTM branch unchanged, i.e., without any rotation-based normalisation.…”

“…This section will introduce the main contribution of this paper, namely the alternative pre-processing techniques that can be used on the separable STA introduced by Das et al [ 17 ], which can improve recognition of actions on the Toyota Smarthomes dataset. On the one hand, skeletal data, which is fed to the LSTM brach, will be normalised, rotating the skeletons so that they ‘look at’ the camera (rotation among the Y axis).…”

“…In the Toyota Smarthomes dataset [ 17 ], skeletal data is not obtained from an RGB-D sensor, as done in other activity recognition datasets, such as the NTU dataset [ 15 , 16 ]. Rather, an LCR-Net (Localization-Classification-Regression) neural network [ 22 ] is applied to the RGB video clips for each action, thus obtaining skeletons for each frame.…”

“…The original paper of Das et al [ 17 ] does video clip pre-processing by cropping the area around the detection of single shot multi-box detector (SSD) network model [ 24 ]. However, this has two disadvantages: first, this is a box detector, rather than a mask detector; therefore, crops might not be as accurate; and second, this is not a “historical” cropping, i.e., taking into consideration dynamics of the detected person throughout time.…”