A Test-Retest Reliability Generalization Meta-Analysis of Judgments Via the Policy-Capturing Technique

Abstract

Policy capturing is a widely used technique, but the temporal stability of policy-capturing judgments has long been a cause for concern. This article emphasizes the importance of reporting reliability, and in particular test-retest reliability, estimates in policy-capturing studies. We found that only 164 of 955 policy-capturing studies (i.e., 17.17%) reported a test-retest reliability estimate. We then conducted a reliability generalization meta-analysis on policy-capturing studies that did report test-retest reliability estimates—and we obtained an average reliability estimate of .78. We additionally examined 16 potential methodological and substantive antecedents to test-retest reliability (equivalent to moderators in validity generalization studies). We found that test-retest reliability was robust to variation in 14 of the 16 factors examined but that reliability was higher in paper-and-pencil studies than in web-based studies and was higher for behavioral intention judgments than for other (e.g., attitudinal and perceptual) judgments. We provide an agenda for future research. Finally, we provide several best-practice recommendations for researchers (and journal reviewers) with regard to (a) reporting test-retest reliability, (b) designing policy-capturing studies for appropriate reportage, and (c) properly interpreting test-retest reliability in policy-capturing studies.

Test-Retest Reliability in Metric Conjoint Experiments: A New Workflow to Evaluate Confidence in Model Results

“…The average test-retest reliability across all studies is r = .78, with a standard deviation of 0.09. This finding is comparable to Zhu et al (2022), who report a mean r = .78 for policy-capturing designs in the broader management literature. Only seven studies report a test-retest reliability below the r .…”

“…Many factors can influence the response consistency of a test score, such as memory effects (Tourangeau et al, 2000), transient errors, and random response behavior (Schmidt & Hunter, 1999). In conjoint studies, test-retest reliability is often used as an indicator for response consistency (Lohrke et al, 2010), where researchers usually interpret a low reliability metric as indicating: (a) the respondent not engaging with the instrument across multiple periods; (b) the instrument itself may be problematic; (c) the decision situation could be unclear; or (d) some combination of all these factors (Karren & Barringer, 2002;Zhu et al, 2022).…”

“…To assess response stability, researchers frequently employ a test-retest reliability metric under the conventional assumption that a higher reliability coefficient-typically r . .70-equates to an acceptable degree of response stability (Karren & Barringer, 2002;Zhu et al, 2022).…”

Qualitative Studies in Management Research: An Emerging Epistemology of Meta-Analysis

“…Variance-based case studies: This is an early and fairly well-propagated genre, pioneered essentially by Eisenhardt [40,41] in the late nineties. Here, constructs are studied and then a relationship between them is sought so as to permit a higher-level knowledge to emerge, mostly to seek for causality [42,43]. Multiple case studies can be replicated and data sets compared for pattern [44].…”

Unraveling leadership potential

“…These alphas are higher compared with values in prior research (Bouland-van Dam et al, 2022). The test-retest correlation of the LAS was .72, p < .01, and ranged from .65 to .71 (all p's < .01) for the dimensions, which is considered satisfactory (DeVellis, 2017;Zhu et al, 2021).…”