Framework for Selecting and Benchmarking Mobile Devices in Psychophysiological Research

Abstract

Commercially available consumer electronics (smartwatches and wearable biosensors) are increasingly enabling acquisition of peripheral physiological and physical activity data inside and outside of laboratory settings. However, there is scant literature available for selecting and assessing the suitability of these novel devices for scientific use. To overcome this limitation, the current paper offers a framework to aid researchers in choosing and evaluating wearable technologies for use in empirical research. Our seven-step framework includes: (1) identifying signals of interest; (2) characterizing intended use cases; (3) identifying study-specific pragmatic needs; (4) selecting devices for evaluation; (5) establishing an assessment procedure; (6) performing qualitative and quantitative analyses on resulting data; and, if desired, (7) conducting power analyses to determine sample size needed to more rigorously compare performance across devices. We illustrate the application of the framework by comparing electrodermal, cardiovascular, and accelerometry data from a variety of commercial wireless sensors (Affectiva Q, Empatica E3, Empatica E4, Actiwave Cardio, Shimmer) relative to a well-validated, wired Mindware laboratory system. Our evaluations are performed in two studies (N=10, N=11) involving psychometrically sound, standardized tasks that include physical activity and affect induction. After applying our framework to this data, we conclude that only some commercially available consumer devices for physiological measurement are capable of wirelessly measuring peripheral physiological and physical activity data of sufficient quality for scientific use cases. Thus, the framework appears to be beneficial at suggesting steps for conducting more systematic, transparent, and rigorous evaluations of mobile physiological devices prior to deployment in studies.

A Usability Study of Physiological Measurement in School Using Wearable Sensors

“…Most of HR samples fall between resting-state HR of European adolescents (77.4 and 81.1 bpm for boys and girls, respectively [ 35 ]) and HR during moderate movement (97 and 101 bpm for boys and girls, respectively calculated by averaging between standing rest-state HR and the lowest exercising HR [ 36 ]). The EDA of participants (mean = 15.98, SD = 7.21 S) was considerably higher than found using wearable EDA devices recording at the wrist with dry electrodes [ 7 , 37 ] which can be explained by the higher density of sweat glands at the palm [ 25 ]. The EDA panel Figure 2 shows a conspicuous peak on the left reflecting a high number of EDA signals lower than 0.5 S. The is likely caused by the loss of attachment between electrode and palmar skin.…”

Electrodermal activity patient simulator

“…It is more and more used in different areas of research, from medicine, ergonomics, biomedical and control engineering, robotics and ergonomics, psychology and education, sports, entertainment to social science and economics. But it is never seriously doubted, in the sense of determining measuring accuracy and quality of measuring result like in [38][39][40]. In this paper a concept of an EDA simulator for testing EDA devices is, to the best of our knowledge, presented for the first time.…”

Evaluating commercially available wireless cardiovascular monitors for measuring and transmitting real‐time physiological responses in children with autism

“…Results also further substantiate the approach of testing new consumer‐grade wearables as they enter the market, which is essential as the market is in constant flux, with new wearables being released at a speed faster than typical research timelines. Therefore, the metrics used in both studies, Sampling Fidelity and Spike Rate, can be applied to new heart rate trackers to assess their fit for future research projects (for additional testing resources see Kleckner et al, 2021).…”

“…However, research involving systematic testing of wearable physiological recording monitors is needed for the field to move toward real‐world applications with confidence (Koumpouros & Kafazis, 2019). Although there is a basic agreement for the need to perform certain types of analyses, such as measures of data loss (see Kleckner et al, 2021), consensus on a methodology to establish this technology's viability for scientific use is lacking in specificity, that is, which quality metrics and thresholds to evaluate these monitors against.…”