Avoiding Bias in Classification Accuracy - A Case Study for Activity Recognition

Abstract

The amount of studies on classification of human characteristics based on measured individual signals has increased rapidly. In wearable sensors based activity recognition a common policy is to report human independent recognition results using leave-one-person-out cross-validation scheme. This can be a suitable solution when feature or model parameter selection is not needed or it is done outside the validation scheme. Unfortunately, this is not always the reality. Thus in this article it is studied how the train-validate-test approach changes the recognition rates compared to basic leave-one-out cross-validation approach. Results of three different ways to perform the trainvalidate-test is presented: 1) single division to training and testing data, 2) 10-fold division to training and testing data, and 3) double leave-one-person-out cross-validation. In this article, it is shown that the best classifier or feature set selected based on the training and validation data using basic leave-one-out approach does not always perform best within independent testing data. Nevertheless, a larger bias to results can be achieved using single division or even 10-fold division into separate training and testing data. Thus it is stated that the double leave-one-person-out is the most robust version for reporting classification rates in future studies of activity recognition as well as other areas where human signals are used.

Accelerometer vs. Electromyogram in Activity Recognition

“…For mRMR no information of the actual problem was introduced. Nevertheless, it has been already show that the recognition rates are biased in SFS while the same data is used for selecting the features and validating the features (Koskimäki, 2015). Thus by using SFS the unseen activities are not unseen but already used in model optimization.…”

“…Both of the methods are fast to train, easy to implement and the memory requirements are small thus making them well-liked in practical applications and devices. Moreover, it has been shown in practical activity recognition applications the simplest methods can outperform the more sophisticated methods (Koskimäki, 2015).…”

Workplace activity classification from shoe-based movement sensors

“…Validation of the classification algorithms was performed by leave-one-out validation for each subject. This validation technique implies that there will never be data from the same subject in both the training- and validation data set simultaneously which leads to a more realistic accuracy measure [ 24 ].

Fig.

…”

Using Sleep Time Data from Wearable Sensors for Early Detection of Migraine Attacks

“…In the case of user-independent models, the over-fitting was easily avoided by using leave-one-out cross-validation: in turn, one person’s data were used as independent testing data and data from the other six study subjects were used for validation and training. The protocol used in the recognition process is shown in Figure 2 [ 27 ].…”