A standardized validity assessment protocol for physiological signals from wearable technology: Methodological underpinnings and an application to the E4 biosensor

Abstract

Wearable physiological measurement devices for ambulatory research with novel sensing technology are introduced with ever increasing frequency, requiring fast, standardized, and rigorous validation of the physiological signals measured by these devices and their derived parameters. At present, there is a lack of consensus on a standardized protocol or framework with which to test the validity of this new technology, leading to the use of various (often unfit) methods. This study introduces a comprehensive validity assessment protocol for physiological signals (electrodermal activity and cardiovascular activity) and investigates the validity of the E4 wearable (an example of such a new device) on the three levels proposed by the protocol: (1) the signal level, with a crosscorrelation; (2) the parameter level, with Bland-Altman plots; and (3) the event level, with the detection of physiological changes due to external stressor levels via event difference plots. The results of the protocol show that the E4 wearable is valid for heart rate, RMSSD, and SD at the parameter and event levels, and for the total amplitude of skin conductance responses at the event level when studying strong sustained stressors. These findings are in line with the prior literature and demonstrate the applicability of the protocol. The validity assessment protocol proposed in this study provides a comprehensive, standardized, and feasible method for assessment of the quality of physiological data coming from new wearable (sensor) technology aimed at ambulatory research.

A Usability Study of Physiological Measurement in School Using Wearable Sensors

“…Specifically, fingers and palm have a higher density of sweat glands compared to wrist causing differences in level of SCL, SCR, range, and sensitivity in response to arousing events [ 25 ]. EDA levels recorded from the wrist-type sensor were found to be typically around the lower bound of valid EDA range (0.05–60 S) [ 7 ], making it hard to detect small EDA responses [ 59 ] and resulting in limited performance in detecting short mild stressors [ 60 ], especially when comparing with the performance of palm-based EDA device [ 61 ]. For a rule-based approach, it is necessary to establish specific criteria that match expert ratings [ 41 ], which may not generalize across signals captured from different devices, contexts, or study goals.…”

Framework for Selecting and Benchmarking Mobile Devices in Psychophysiological Research

“…Data quality can be assessed using signal-to-noise ratio; measures of data loss (e.g., missing heart beats); or simple measures of agreement such as Pearson product-moment correlations, intraclass correlations, or Bland-Altman analyses (Bland & Altman, 2007; for example decision criterion see van Lier 2019). When assessing both general trends and data quality, we endorse recently published guidelines which suggest assessment at the signal level (e.g., raw skin conductance), the parameter level (e.g., rate of skin conductance responses), and the event level (e.g., rate of skin conductance responses during lower arousal vs. higher arousal scenarios) (van Lier et al, 2019). Finally, qualitative data from user-feedback forms can be explored (e.g., using thematic coding, simple statistics, or visualization) to unearth participant concerns in addition to any individual differences which may have led to usability problems (for examples, see Beukenhorst et al, 2020;Shcherbina et al, 2017).…”

“…Extant validation studies typically limit their assessment to one of these two categories, either focusing exclusively on user-experience (e.g., Beaukenhorst et al 2020) or on signal quality. Additionally, those that focus on signal quality tend to emphasize either qualitative measures (e.g., McCarthy et al, 2016) or quantitative measures (e.g., Kasos et al, 2019;Straiton et al, 2018;van Lier et al, 2019;Weippert et al, 2010). While each of these categories of validation are informative and useful in their own right, variability in approaches can be intimidating for newcomers interested in utilizing ambulatory measurement in their research.…”

“…While science benefits from published guidelines, they too can be limited in scope by focusing on specific signals, statistical analytic methods, or analytic decision criteria (Parati et al, 2010(Parati et al, , 2014van Lier et al, 2019). Rarely is one set of criteria sufficient for establishing validity and utility in science.…”

“…van Lier et al, 2019) and validation studies (e.g., Kasos et al, 2019;van Lier et al, 2019) for assessing mobile devices by including considerations for the broader set of decisions that researchers must make prior to initiating a study. That is, our framework emphasizes selection of mobile devices based on signals of interest, intended use cases, pragmatic needs (Steps 1-4), and establishes an effective assessment procedure to test the strengths and limitations of selected devices (Step 5).…”