Sample Size for Cognitive Interview Pretesting

Blair, Johnny; Conrad, Frederick G.

doi:10.1093/poq/nfr035

Cited by 115 publications

(97 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Cognitive interviewing as a methodological approach cannot indicate the size or extent of a problem with particular items on the questionnaire, nor can it guarantee that all problems have been captured, especially as research suggests there is a positive relationship between sample size and problem detection. 16 Using cognitive interviewing techniques in a longitudinal study design may have led to participants becoming 'schooled' in the questionnaire. The use of 'nondirective' and 'observational probes' during questionnaire completion may have influenced how responders continued to map their answers.…”

Section: Strengths and Limitationsmentioning

confidence: 99%

Usefulness of PHQ-9 in primary care to determine meaningful symptoms of low mood: a qualitative study

et al. 2016

View full text Add to dashboard Cite

show abstract

Section: Strengths and Limitationsmentioning

confidence: 99%

Usefulness of PHQ-9 in primary care to determine meaningful symptoms of low mood: a qualitative study

et al. 2016

View full text Add to dashboard Cite

show abstract

“…This means that the researcher may select a purposeful sample of individuals who are likely to encounter problems with the proposed questions, such as cultural minorities, less educated persons, or people with low cognitive abilities. Related to this, the researcher may seek to increase the detectability of problems [13], through the use of various in-depth cognitive interviewing methods [1,6], and also by selecting selfaware or introspective participants who have a greater capacity to reflect on their thinking patterns. If the detectable prevalence of the problem can be shifted from 5 %, which may be easily missed in a small sample, to 10 or 15 %, the pre-test will be considerably more productive.…”

Section: Discussionmentioning

confidence: 99%

“…If the sample size is too small, the probability can be large that no participant will report any given problem. A recent paper by Blair and Conrad explores this issue [13]. These authors show that sample size requirements are often much larger than the customary numbers cited above, for various values of problem prevalence, probability of detection, and power.…”

Section: Introductionmentioning

confidence: 99%

Sample size for pre-tests of questionnaires

et al. 2014

View full text Add to dashboard Cite

Purpose To provide guidance regarding the desirable size of pre-tests of psychometric questionnaires, when the purpose of the pre-test is to detect misunderstandings, ambiguities, or other difficulties participants may encounter with instrument items (called «problems»).Methods We computed (a) the power to detect a problem for various levels of prevalence and various sample sizes, (b) the required sample size to detect problems for various levels of prevalence, and (c) upper confidence limits for problem prevalence in situations where no problems were detected. Results As expected, power increased with problem prevalence and with sample size. If problem prevalence was 0.05, a sample of 10 participants had only a power of 40 % to detect the problem, and a sample of 20 achieved a power of 64 %. To achieve a power of 80 %, 32 participants were necessary if the prevalence of the problem was 0.05, 16 participants if prevalence was 0.10, and 8 if prevalence was 0.20. If no problems were observed in a given sample, the upper limit of a two-sided 90 % confidence interval reached 0.26 for a sample size of 10, 0.14 for a sample size of 20, and 0.10 for a sample of 30 participants.Conclusions Small samples (5-15 participants) that are common in pre-tests of questionaires may fail to uncover even common problems. A default sample size of 30 participants is recommended.

show abstract

“…It is also possible that sampling a small number of participants will not represent the realm of experiences. Indeed, Blair and Conrad (2011) found that when conducting cognitive pretesting for the purpose of informing score validity, the probability of detecting issues with a survey increased as the number of participants increased. Literature on determining sample sizes for interviewing provides further guidance on sampling (e.g., Creswell, 2007;Graves, 2002;Guest, Bunce, & Johnson, 2006).…”

Section: Example Illustrationmentioning

confidence: 99%

Using the Cognitive Pretesting Method to Gain Insight Into Participants’ Experiences

Koskey

2016

International Journal of Qualitative Methods

View full text Add to dashboard Cite

Quantitative researchers traditionally use cognitive pretesting methods such as think-alouds to determine whether participants are interpreting close-ended survey items as intended. I applied this method to evaluate the score validity of quantitative survey instruments with a middle school and high school sample of students. Through these applications, the cognitive pretesting method also revealed to be a useful tool for cueing participants' reflection on their experiences related to the phenomenon of interest. Qualitative researchers traditionally use more open-ended interviews guided by general ''how'' and ''why'' questions, which might not serve as sufficient prompts to elicit participants' reflection on their experiences, especially younger participants. To date, there are no illustrative examples focused on how having participants think-aloud as they complete a quantitative survey can inform qualitative-focused or mixed research questions. Therefore, the purpose of this article was to illustrate how the cognitive pretesting method, where participants think-aloud as they complete a survey, can be used as an additional technique to gain insight into their experiences of a phenomenon. The phenomenon in this illustration was defined as the participants' experiences with the science content that they were learning in school. Limitations and methodological challenges and recommendations for overcoming these barriers are also discussed.

show abstract

Sample Size for Cognitive Interview Pretesting

Cited by 115 publications

References 12 publications

Usefulness of PHQ-9 in primary care to determine meaningful symptoms of low mood: a qualitative study

Usefulness of PHQ-9 in primary care to determine meaningful symptoms of low mood: a qualitative study

Sample size for pre-tests of questionnaires

Using the Cognitive Pretesting Method to Gain Insight Into Participants’ Experiences

Contact Info

Product

Resources

About