Performance Evaluation of State of the Art Systems for Physical Activity Classification of Older Subjects Using Inertial Sensors in a Real Life Scenario: A Benchmark Study

Abstract

The popularity of using wearable inertial sensors for physical activity classification has dramatically increased in the last decade due to their versatility, low form factor, and low power requirements. Consequently, various systems have been developed to automatically classify daily life activities. However, the scope and implementation of such systems is limited to laboratory-based investigations. Furthermore, these systems are not directly comparable, due to the large diversity in their design (e.g., number of sensors, placement of sensors, data collection environments, data processing techniques, features set, classifiers, cross-validation methods). Hence, the aim of this study is to propose a fair and unbiased benchmark for the field-based validation of three existing systems, highlighting the gap between laboratory and real-life conditions. For this purpose, three representative state-of-the-art systems are chosen and implemented to classify the physical activities of twenty older subjects (76.4 ± 5.6 years). The performance in classifying four basic activities of daily life (sitting, standing, walking, and lying) is analyzed in controlled and free living conditions. To observe the performance of laboratory-based systems in field-based conditions, we trained the activity classification systems using data recorded in a laboratory environment and tested them in real-life conditions in the field. The findings show that the performance of all systems trained with data in the laboratory setting highly deteriorates when tested in real-life conditions, thus highlighting the need to train and test the classification systems in the real-life setting. Moreover, we tested the sensitivity of chosen systems to window size (from 1 s to 10 s) suggesting that overall accuracy decreases with increasing window size. Finally, to evaluate the impact of the number of sensors on the performance, chosen systems are modified considering only the sensing unit worn at the lower back. The results, similarly to the multi-sensor setup, indicate substantial degradation of the performance when laboratory-trained systems are tested in the real-life setting. This degradation is higher than in the multi-sensor setup. Still, the performance provided by the single-sensor approach, when trained and tested with real data, can be acceptable (with an accuracy above 80%).

Physical Activity Classification for Elderly People in Free-Living Conditions

“…Our earlier work [25] focused on this issue by highlighting the gaps and limitations imposed by free living conditions on existing PAC systems which have been developed in laboratory-based environments. To provide an unbiased and fair comparison we worked to have full control over the nature of the dataset (set of ADLs, studied population), sampling frequency, window size and cross validation procedure.…”

Performance Analysis of Boosting Classifiers in Recognizing Activities of Daily Living

“…Moreover, the impact of feature selection methods on the performance of boosting classifiers is not studied systematically considering the domain of PAC, and very little is known about how these classifiers behave when the feature selection stage is incorporated before classification. To better address this issue, a benchmark analysis was carried out by Awais et al [35], which provides the sequence of steps that can be performed to provide a balanced and unbiased performance analysis of different PAC systems using a different type of classifiers. This study also investigates the performance of recently developed Catboost classifiers for PAC.…”

“…Although the change (increase or decrease) in the performance was not quite significant as compared with the performances achieved through the whole feature set, the number of features was significantly reduced from 561 to 150 using the CFS approach (over 70% reduction in several features). The number of features has implications on the computational complexity of the system [35,37]. A large number of features increases the computational complexity of the system and makes the systems infeasible to operate in real-time scenarios.…”

A Survey on Modality Characteristics, Performance Evaluation Metrics, and Security for Traditional and Wearable Biometric Systems

“…Based on these factors, many performance evaluation metrics have been proposed recently in literature, including some patents [32,34,121,182]. They have all been investigated for healthcare and personal fitness, specifically for PPG.…”