A catalog of validity indices for step counting wearable technologies during treadmill walking: the CADENCE-adults study

Abstract

Background Standardized validation indices (i.e., accuracy, bias, and precision) provide a comprehensive comparison of step counting wearable technologies. Purpose To expand a previously published child/youth catalog of validity indices to include adults (21–40, 41–60 and 61–85 years of age) assessed across a range of treadmill speeds (slow [0.8–3.2 km/h], normal [4.0–6.4 km/h], fast [7.2–8.0 km/h]) and device wear locations (ankle, thigh, waist, and wrist). Methods Two hundred fifty-eight adults (52.5 ± 18.7 years, 49.6% female) participated in this laboratory-based study and performed a series of 5-min treadmill bouts while wearing multiple devices; 21 devices in total were evaluated over the course of this multi-year cross-sectional study (2015–2019). The criterion measure was directly observed steps. Computed validity indices included accuracy (mean absolute percentage error, MAPE), bias (mean percentage error, MPE), and precision (correlation coefficient, r; standard deviation, SD; coefficient of variation, CoV). Results Over the range of normal speeds, 15 devices (Actical, waist-worn ActiGraph GT9X, activPAL, Apple Watch Series 1, Fitbit Ionic, Fitbit One, Fitbit Zip, Garmin vivoactive 3, Garmin vivofit 3, waist-worn GENEActiv, NL-1000, PiezoRx, Samsung Gear Fit2, Samsung Gear Fit2 Pro, and StepWatch) performed at < 5% MAPE. The wrist-worn ActiGraph GT9X displayed the worst accuracy across normal speeds (MAPE = 52%). On average, accuracy was compromised across slow walking speeds for all wearable technologies (MAPE = 40%) while all performed best across normal speeds (MAPE = 7%). When analyzing the data by wear locations, the ankle and thigh demonstrated the best accuracy (both MAPE = 1%), followed by the waist (3%) and the wrist (15%) across normal speeds. There were significant effects of speed, wear location, and age group on accuracy and bias (both p < 0.001) and precision (p ≤ 0.045). Conclusions Standardized validation indices cataloged by speed, wear location, and age group across the adult lifespan facilitate selecting, evaluating, or comparing performance of step counting wearable technologies. Speed, wear location, and age displayed a significant effect on accuracy, bias, and precision. Overall, reduced performance was associated with very slow walking speeds (0.8 to 3.2 km/h). Ankle- and thigh-located devices logged the highest accuracy, while those located at the wrist reported the worst accuracy. Trial registration Clinicaltrials.gov NCT02650258. Registered 24 December 2015.

Accuracy and Reliability of a Suite of Digital Measures of Walking Generated Using a Wrist-worn Sensor: Performance Characterization Study

“…Due to the known potential errors associated with body placement when capturing walking-related data [36][37][38][39], co-location could be vulnerable to bias towards overestimating performance, which our approach seeks to mitigate. Further, most studies have had a narrow focus on step counts [10][11][12][13][14][15][16], mostly in controlled laboratory environments and/or for limited time periods (e.g., single day in real-world setting). Thus, directly comparing study results side by side has to be done with caution, given the heterogeneity of the studies (comparisons of different devices, different ground truth sources, and with different analytic approaches), which highlights the need for standardization noted in professional statements in this field [6,9,40,41].…”

“…However, the literature around these studies shows considerable heterogeneity across designs and some notable limitations. First, analyses tend to rely on truth labels originated by participants' self reports, short-term close monitoring [10][11][12], or from reference devices with suboptimal accuracy (mean absolute percentage error >20%) and/or with the same body placement as the investigational devices, which would bias agreement results [13]. Second, these studies are often conducted in artificial laboratory environments, which inherently limit behavior range and are susceptible to subjectivity, assessment bias and/or unreliability [14][15][16].…”

Physical activity and sleep changes among children during the COVID-19 pandemic

“…The Garmin VivoFit 3 has been reported to accurately monitor step counts in children through multiple research studies, which demonstrated, for example, equivalence to pedometer-measured step counts 63 , 64 . The Garmin VivoFit 3 has also been shown to perform at <5% mean absolute percentage error for laboratory-based treadmill study over the range of normal speeds in adults 65 . Participants were instructed to wear the watch 24 h a day for the entire one-year study duration.…”

Incorporating Wearable Technology for Enhanced Rehabilitation Monitoring after Hip and Knee Replacement

“…It is noteworthy, however, that the device tends to underestimate step counts, especially at slower walking speeds and when individuals are engaged in indoor walking with frequent postural transitions [ 29 ]. The selection of the Garmin Vivofit ® 4 for our data collection process was informed by its recognized reliability in capturing physical activity metrics, aligning with the specific considerations highlighted in the literature [ 30 ].…”