Fully Convolutional Network Bootstrapped by Word Encoding and Embedding for Activity Recognition in Smart Homes

Abstract

Activity recognition in smart homes is essential when we wish to propose automatic services for the inhabitants. However, it is a challenging problem in terms of environments' variability, sensory-motor systems, user habits, but also sparsity of signals and redundancy of models. Therefore, end-to-end systems fail at automatically extracting key features, and need to access context and domain knowledge. We propose to tackle feature extraction for activity recognition in smart homes by merging methods of Natural Language Processing (NLP) and Time Series Classification (TSC) domains.We evaluate the performance of our method with two datasets issued from the Center for Advanced Studies in Adaptive Systems (CASAS). We analyze the contributions of the use of embedding based on term frequency encoding, to improve automatic feature extraction. Moreover we compare the classification performance of Fully Convolutional Network (FCN) from TSC, applied for the first time for activity recognition in smart homes, to Long Short Term Memory (LSTM). The method we propose, shows good performance in offline activity classification. Our analysis also shows that FCNs outperforms LSTMs, and that domain knowledge gained by event encoding and embedding improves significantly the performance of classifiers.

A Survey of Human Activity Recognition in Smart Homes Based on IoT Sensors Algorithms: Taxonomies, Challenges, and Opportunities with Deep Learning

“…However, in recent years, researchers in the field of Natural Language Processing (NLP) have developed techniques of word embedding and the language model for deep learning algorithms to understand not only the meaning of words but also the structure of phases and texts. A first attempt to add NLP word embedding to deep learning has shown a better performance in daily activity recognition in smart homes [46]. Moreover, the use of the semantics of the HAR domain may allow the development of new learning techniques for quick adaptation, such as zero-shot learning, which is developed in Section 5.…”

“…Applied to HAR, they could model the context of the sensors and their order of appearance. Taking inspiration from References [46,66], we can draw a parallel between NLP and HAR: a word is apparent to a sensor event, a micro activity composed of sensor events is apparent to a sentence, and a compound activity composed of sub-activities is a paragraph. The parallel between word and sensor events has led to the combination of word encodings with deep learning to improve the performance of HAR in smart homes in Reference [46].…”

“…For instance, sparsity of the data can be alleviated by a better domain knowledge in the form of an emerging semantic. Thus, taking inspiration from word encoding and language models, we can automatically introduce semantic knowledge between activities, as shown in the preliminary study of Reference [46]. Furthermore, a semantic encoding of the data will also help the system be more robust to unknown data as in the challenges of data drift or adaptation to changes, as it could be able to relate new data semantically to known data.…”

An Analysis of Image Features Extracted by CNNs to Design Classification Models for COVID-19 and Non-COVID-19

“…Regarding the EB0 method, the returned vector has 9408 dimensions; • GlobalAveragePooling2D: this parameter represents a global pooling operation for spatial representation of the data, which is applied to replace the stage of fully connected layers in conventional CNNs. This operation generates a resource map for each class that will be classified [84]. This operation was used to obtain a drastic reduction of the dimensions of the feature maps in the layers, top_activation, mixed9 and mixed10 of the respective methods of the algorithms: EfficientNetB0 (EB0 baseline) and InceptionV3 (IV3 and IV3 baseline).…”

Deep Learning for Human Activity Recognition Based on Causality Feature Extraction

“…Some of the challenges are irrelevant and routine activities, whereas others are related to a predefined target class. Unlabeled data can significantly reduce the AR performance [53][54][55][56] because of their correlation with labeled data. In real-time AR services that test all sensor data generated in real time in order, if the AR model is not trained on the other data, it will erroneously recognize queries as predefined activities even though there is no correspondence between them.…”