Investigating (re)current state-of-the-art in human activity recognition datasets

Abstract

Many human activities consist of physical gestures that tend to be performed in certain sequences. Wearable inertial sensor data have as a consequence been employed to automatically detect human activities, lately predominantly with deep learning methods. This article focuses on the necessity of recurrent layers—more specifically Long Short-Term Memory (LSTM) layers—in common Deep Learning architectures for Human Activity Recognition (HAR). Our experimental pipeline investigates the effects of employing none, one, or two LSTM layers, as well as different layers' sizes, within the popular DeepConvLSTM architecture. We evaluate the architecture's performance on five well-known activity recognition datasets and provide an in-depth analysis of the per-class results, showing trends which type of activities or datasets profit the most from the removal of LSTM layers. For 4 out of 5 datasets, an altered architecture with one LSTM layer produces the best prediction results. In our previous work we already investigated the impact of a 2-layered LSTM when dealing with sequential activity data. Extending upon this, we now propose a metric, rGP, which aims to measure the effectiveness of learned temporal patterns for a dataset and can be used as a decision metric whether to include recurrent layers into a network at all. Even for datasets including activities without explicit temporal processes, the rGP can be high, suggesting that temporal patterns were learned, and consequently convolutional networks are being outperformed by networks including recurrent layers. We conclude this article by putting forward the question to what degree popular HAR datasets contain unwanted temporal dependencies, which if not taken care of, can benefit networks in achieving high benchmark scores and give a false sense of overall generability to a real-world setting.

Hang-Time HAR: A Benchmark Dataset for Basketball Activity Recognition Using Wrist-Worn Inertial Sensors

“…According to Bock et al [106] we distinguish between sporadic, simple/periodical, transitional, and complex activities. However, datasets shown in Table 1 mostly focus on locomotion activities and activities of daily living.…”

“…However, datasets shown in Table 1 mostly focus on locomotion activities and activities of daily living. Only a few, such as [26,27,35,51], include sporadic, transition or complex activities, and many datasets that do include sports [16,33] aggregate an entire sport into a single activity. Published sports studies tend to not release their datasets publicly or only upon request-with Trost et al [17] and Bock et al [51] as the only exceptions, as shown in Tables 2 and 3.…”

“…Only a few, such as [26,27,35,51], include sporadic, transition or complex activities, and many datasets that do include sports [16,33] aggregate an entire sport into a single activity. Published sports studies tend to not release their datasets publicly or only upon request-with Trost et al [17] and Bock et al [51] as the only exceptions, as shown in Tables 2 and 3. As a result, sports-specific IMU-based datasets available to the public that reflect the complexity and characteristics of a specific sport are very limited.…”

“…IMU-based sports activity recognition is one of the main application fields for HAR studies, as summarized in Tables 2 and 3. It has already been proven for a variety of different sports, such as running [37][38][39], ball sports [36,[40][41][42][43][44], winter sports [45,46], sports for the disabled [47], or fitness [48][49][50][51], that activity recognition algorithms are capable of detecting specific activities tied to these sports based on IMU data as input. Other studies that incorporated IMUs focus rather on the athletes' performance [52][53][54][55] or gait estimation [56,57] than activity recognition.…”

“…Employing the definition as defined in [106], Hang-Time HAR offers both complex (shot, layup) and sporadic (rebound, pass) activities. As these activities are not as reliably detected (even in controlled an environment) as other activities, it is to be investigated whether this lies in the nature of the activities, or can be accredited to the employed network architecture reaching its limits.…”