Continual Activity Recognition with Generative Adversarial Networks

Abstract

Continual learning is an emerging research challenge in human activity recognition (HAR). As an increasing number of HAR applications are deployed in real-world environments, it is important and essential to extend the activity model to adapt to the change in people’s activity routine. Otherwise, HAR applications can become obsolete and fail to deliver activity-aware services. The existing research in HAR has focused on detecting abnormal sensor events or new activities, however, extending the activity model is currently under-explored. To directly tackle this challenge, we build on the recent advance in the area of lifelong machine learning and design a continual activity recognition system, called HAR-GAN , to grow the activity model over time. HAR-GAN does not require a prior knowledge on what new activity classes might be and it does not require to store historical data by leveraging the use of Generative Adversarial Networks (GAN) to generate sensor data on the previously learned activities. We have evaluated HAR-GAN on four third-party, public datasets collected on binary sensors and accelerometers. Our extensive empirical results demonstrate the effectiveness of HAR-GAN in continual activity recognition and shed insight on the future challenges.

Lifelong Adaptive Machine Learning for Sensor-based Human Activity Recognition Using Prototypical Networks

“…CL-HAR-1 [15] CL-HAR-2 [16] EMILEA [18] HAR-GAN [19] LAPNet-HAR (ours) recognition as well as in handling the catastrophic forgetting and model intransigence tradeoff across all datasets. This further uncovered findings in dealing with various challenges of task-free continual learning in HAR.…”

“…[18] proposed EMILEA, a technique to evolve an activity model over time by integrating dynamic network expansion Net2Net [30] for enhancing the model's capacity with increasing number of activities and Gradient Episodic Memory [31] for mitigating the effect of catastrophic forgetting. They further proposed another continual activity recognition system, HAR-GAN, that leverages Generative Adversarial Networks (GAN) to generate sensor data on previous classes without the need to store historical data [19].…”

“…It is clear that the field is still under-explored and researchers studying continual learning in HAR have barely scratched the surface. To date, proposed methods, such as EMILEA [18] and HAR-GAN [19], have relied on dynamically expanding network architectures with increasing number of classes. While shown effective, this leads to increasing number of model parameters with the growing set of activities, which can be limiting for a deployable resource-constraint solution.…”

Lifelong Adaptive Machine Learning for Sensor-based Human Activity Recognition Using Prototypical Networks

“…[55] proposed EMILEA, a technique to evolve an activity model over time by integrating dynamic network expansion Net2Net [8] for enhancing the model's capacity with increasing number of activities and Gradient Episodic Memory [30] for mitigating the effect of catastrophic forgetting. They further proposed another continual activity recognition system, HAR-GAN, that leverages Generative Adversarial Networks (GAN) to generate sensor data on previous classes without the need to store historical data [56].…”

“…It is clear that the field is still under-explored and researchers studying continual learning in HAR have barely scratched the surface. To date, proposed methods, such as EMILEA [55] and HAR-GAN [56], have relied on dynamically expanding network architectures with increasing number of classes. While shown effective, this leads to increasing number of model parameters with the growing set of activities, which can be limiting for a deployable resource-constraint solution.…”