A large-scale analysis of test-retest reliabilities of self-regulation measures

Abstract

The ability to regulate behavior in service of long-term goals is a widely studied psychological construct known as self-regulation. This wide interest is in part due to the putative relations between self-regulation and a range of real-world behaviors. Self-regulation is generally viewed as a trait, and individual differences are quantified using a diverse set of measures including self-report surveys and behavioral tasks. Accurate characterization of individual differences requires measurement reliability, a property frequently characterized in self-report surveys, but rarely assessed in behavioral tasks. We remedy this gap by (1) providing a comprehensive literature review on an extensive set of self-regulation measures, and (2) empirically evaluating retest reliability in this battery of measures in a new sample. We find that self-report survey measures of self-regulation have high test-retest reliability while measures derived from behavioral tasks do not. This holds both in the literature and in our sample. We confirm that this is due to differences in between-subjects variability. We also compare different types of task measures (e.g., model parameters vs. raw response times) in their suitability as individual difference measures, finding that certain model parameters are as stable as raw measures. Our results provide greater psychometric footing for the study of self-regulation and provide guidance for future studies of individual differences in this domain.

Common mechanisms of executive attention underlie executive function and effortful control in children

“…Our findings suggest that stimulus‐response compatibility tasks, such as the attention networks test, Stroop, flanker, and Simon tasks, may not provide adequate measures of executive attention, EC, and self‐regulation more broadly. This conclusion is supported by recent evidence documenting the poor reliability of cognitive tasks, particularly those using difference scores between conditions, as measures of individual differences in self‐regulation (Enkavi et al, ; Hedge et al, ).…”

“…Thus, an alternative explanation for the current findings is that unexplained variance in the self‐regulation factor reflects method variance related to behavioral ratings (Podsakoff et al ). Lack of convergence between the executive attention and self‐regulation factors may also be representative of broader issues in mapping task‐based measures to behavioral ratings of clinically relevant traits (Cyders & Coskunpinar, ; Enkavi et al, ).…”

“…Nevertheless, our finding that executive attention failed to explain unique variance in parent‐reported ADHD problems beyond that explained by behavioral ratings raises questions about the utility of cognitive assessment in self‐regulation and developmental psychopathology research. Performance on cognitive tasks related to self‐regulation are generally associated with low internal consistency reliability, poor stability, and may be ill‐suited for measuring individual differences, particularly those purported to reflect stable traits (Enkavi et al, ; Hedge et al, ). Despite being more costly and time‐intensive, cognitive assessment may offer limited incremental predictive validity in explaining developmental psychopathology beyond behavioral ratings of self‐regulation (Barkley & Fischer, ; Barkley & Murphy, ; Toplak et al, ).…”

“…However, performance on cognitive tasks has typically exhibited weak to modest correlations with behavioral ratings of EF and EC (Rueda, Posner, et al, ; Samyn et al, ; Simonds et al, ; Ten Eycke & Dewey, ; Toplak, Bucciarelli, Jain, & Tannock, ). A lack of convergence between task‐based and self‐report measures is a problem encountered in the self‐regulation research literature more broadly and has been attributed to the poor psychometric properties of task‐based measures, as well as their inability to reliably measure individual differences (Enkavi et al, ; Hedge, Powell, & Sumner, ).…”

“…Second, the cognitive tasks used to measure EF and executive attention are often characterized by poor reliability, which constrains the strength of the empirical associations that can be measured with other variables (Ghiselli, Campbell, & Zedeck, ; Hedge et al, ). Third, EF tasks are affected by task impurity: individual differences in task performance combines variance from multiple sources beyond the function of interest, including non‐target cognitive processes, task‐specific variance, and measurement error (Enkavi et al, ; Snyder, Miyake, & Hankin, ). Fourth, executive attention has been narrowly defined in the developmental literature (Diamond, ) and measured using single tasks in previous studies (e.g.…”