A Z-score based method for comparing the relative sensitivity of behavioral and physiological metrics including cognitive performance, mood, and hormone levels

Abstract

A method for assessing the relative sensitivity of research metrics is proposed and illustrated by comparing 18 outcome measures from a published study of the cognitive, mood, and hormonal effects of four different levels of stress induced by intense military training. Research on the human response to stress often assesses multiple disparate dependent measures. Selecting the most sensitive is difficult as formal methods to compare varied dependent measures have not been developed. The method first converts the outcome measures into standard scores (z‐scores) and then compares them using analysis of variance to determine whether there are differences in how they assess the impact of graded levels of exposure to stress. The analysis detected various significant interactions in several measures and suggests self‐report mood questionnaires were more sensitive to the stressors present in the study than the cognitive or hormonal measures which were used. These findings support the effectiveness of the z‐score based method as a useful procedure for objectively evaluating the differential sensitivity of various metrics. This method could be useful for research on other independent variables when use of multiple assessment strategies is appropriate. It could be used for evaluating studies yielding conflicting results, such as those detecting effects on one parameter but not others. In such instances, cross‐metric inconsistencies may be due to differential sensitivity of measurement strategies rather than actual differences in the effects of the independent‐variable on the domains under investigation.

Age-related changes to the attentional modulation of temporal binding

“…The RTs were then pooled and a grand mean was calculated and used for further analysis. As RTs are known to frequently deviate from normality (Whelan, 2008 ), the grand means were converted into z -scores, following the procedures recommended by Caldwell et al ( 2019 ). A 2 (age: younger vs older) × 2 (cue: valid vs invalid) mixed ANOVA was then conducted on the z -score reaction times produced in the unisensory visual-only conditions, and a 2 (age: younger vs older) × 2 (cue: valid vs invalid) × 3 (SOA: 0 ms × 150 ms × 300 ms) mixed ANOVA was conducted on the z -score reaction times produced in the audiovisual multisensory conditions.…”

“…In addition, the proportion of “bounce” responses produced in the audiovisual conditions (SOAs of 0 ms, 150 ms, and 300 ms) were pooled and a grand mean was calculated and used for further analysis. These grand means were converted into z-scores, following the procedures recommended by Caldwell et al ( 2019 ). A 2 (age: younger vs older) × 2 (cue: valid vs invalid) × 3 (SOA: 0 ms vs 150 ms vs 300 ms) mixed ANOVA was then conducted on these standardized z- score data from the multisensory conditions.…”

“…This produced an overall valid and an overall invalid mean percentage of “bounce” responses for each participant. As with the reaction-time data and full bounce/pass data, these percentages were then pooled to allow calculation of the grand mean and subsequently converted to standardized z -scores, following the procedures recommended by Caldwell et al ( 2019 ). Two separate one-way ANOVAs were conducted on this collapsed z -score data (“age” as the between-subjects factor, and valid or invalid as the within-subjects factor) to investigate differences between younger and older adults in the valid condition, and differences between younger and older adults in the invalid condition (Laerd, 2015 ).…”

Age-Related Changes to the Attentional Modulation of Temporal Binding

“…The RTs were then pooled and a grand mean was calculated and used for further analysis. As RTs are known to frequently deviate from normality (Whelan, 2008), the grand means were converted into z-scores, following the procedures recommended by Caldwell et al (2019). A 2 (age: younger vs older) × 2 (cue: valid vs invalid) mixed ANOVA was then conducted on the z-score reaction times produced in the unisensory visual-only conditions, and a 2 (age: younger vs older) × 2 (cue: valid vs invalid) × 3 (SOA: 0 ms × 150 ms × 300 ms) mixed ANOVA was conducted on the z-score reaction times produced in the audiovisual multisensory conditions.…”

“…In addition, the proportion of "bounce" responses produced in the audiovisual conditions (SOAs of 0 ms, 150 ms, and 300 ms) were pooled and a grand mean was calculated and used for further analysis. These grand means were converted into z-scores, following the procedures recommended by Caldwell et al (2019). A 2 (age: younger vs older) × 2 (cue: valid vs invalid) × 3 (SOA: 0 ms vs 150 ms vs 300 ms) mixed ANOVA was then conducted on these standardized z-score data from the multisensory conditions.…”