Stability of psychometric questionnaires

Abstract

In 1999, Wilson and Batterham proposed a new approach to assessing the test-retest stability of psychometric questionnaires. They recommended assessing the proportion of agreement - that is, the proportion of participants that record the same response to an item - using a test-retest design. They went on to use a bootstrapping technique to estimate the uncertainty of the proportion of agreement. The aims of this short communication are (1) to demonstrate that the sampling distribution of the proportion of agreement is well known (the binomial distribution), making the technique of 'bootstrapping' redundant, and (2) to suggest a much simpler, more sensitive method of assessing the stability of a psychometric questionnaire, based on the test-retest differences (within-individuals) for each item. Adopting methods similar to Wilson and Batterham, 97 sport students completed the Social Physique Anxiety Scale on two occasions. Test-retest differences were calculated for each item. Our results show that the proportion of agreement ignores the nature of disagreement. Items 4 and 11 showed similar agreement (44.3% and 43.3% respectively), but 89 of the participants (91.8%) differed by just +/-1 point when responding to item 4, indicating a relatively stable item. In contrast, only 78 of the participants (80.4%) recorded a difference within +/- 1 point when responding to item 11, suggesting quite contrasting stability for the two items. We recommend that, when assessing the stability of self-report questionnaires using a 5-point scale, most participants (90%) should record test-retest differences within a reference value of +/- 1.

Cross-cultural adaptation and assessment of the reliability and validity of the Core Outcome Measures Index (COMI) for the Brazilian-Portuguese language

“…Differences in response to each domain on the COMI were ±1 category in 98% patients for the domain 'function', 96% for 'symptom-specific well-being', 97% for 'general quality of life', 99% for 'social disability' and 100% for 'work disability', hence all satisfying the stability criterion of C90% suggested by Nevill et al [22]. Table 3 shows the mean (SD) scores on the two test occasions, and the ICC, SEM and MDC95% for each of the scales.…”

“…For the COMI fivepoint ordinal scales, reproducibility (stability) of measures was assessed by examining the proportion of participants recording test-retest differences for each item within a reference value of ±1 point (where at least 90% was considered acceptable) [22,26]. For scales/items with approximately normally distributed interval data (pain scales, COMI summary score, Roland Morris, ODI), oneway repeated measures ANOVA was used to assess the differences in means for the repeated trials and to determine the intraclass correlation coefficient (ICC; model ICC agreement 2,1 ) and their 95% confidence intervals.…”

“…Assuming a similar MCID exists for the Brazilian-Portuguese version, then a clinically relevant change of 2-3 points (the ''signal'') would far exceed the minimum detectable change of 1.7 points (the ''noise''), rendering it an appropriate LBP outcome instrument [13]. The test-retest reliability or ''stability'' of the individual five-point response scale COMI items (function, symptomspecific well-being, quality of life and disability) was assessed using the simple but sensitive method recommended by Nevill et al [22], in which within-individual differences in responses on re-retest are calculated. These authors recommend that, when assessing the stability of five-point scales in self-report questionnaires, 90% participants should record test-retest differences within a reference value of ±1.…”

Confirmatory factor analysis of the Test of Performance Strategies (TOPS) among adolescent athletes

“…Such a strategy may be premature until the stability of the TOPS factor structure is established. Unless it can be demonstrated that scores on a questionnaire are stable under conditions where no change is expected, any changes observed may be caused by random error potentially associated with participants not understanding the meaning of items (see Nevill et al, 2001).…”

Design and validation of an observational instrument to assess the technical execution in top-rope climbing

“…A voluntary informed consent of the beginning climbers' parents was signed previously. The instrument was applied at two different moments with one week of difference (Baumgartner, 2000;Nevil, et al 2001). As usually, the instrument was tested in the same conditions on an indoor climbing wall.…”