Sample size for pre-tests of questionnaires

Abstract

Purpose To provide guidance regarding the desirable size of pre-tests of psychometric questionnaires, when the purpose of the pre-test is to detect misunderstandings, ambiguities, or other difficulties participants may encounter with instrument items (called «problems»).Methods We computed (a) the power to detect a problem for various levels of prevalence and various sample sizes, (b) the required sample size to detect problems for various levels of prevalence, and (c) upper confidence limits for problem prevalence in situations where no problems were detected. Results As expected, power increased with problem prevalence and with sample size. If problem prevalence was 0.05, a sample of 10 participants had only a power of 40 % to detect the problem, and a sample of 20 achieved a power of 64 %. To achieve a power of 80 %, 32 participants were necessary if the prevalence of the problem was 0.05, 16 participants if prevalence was 0.10, and 8 if prevalence was 0.20. If no problems were observed in a given sample, the upper limit of a two-sided 90 % confidence interval reached 0.26 for a sample size of 10, 0.14 for a sample size of 20, and 0.10 for a sample of 30 participants.Conclusions Small samples (5-15 participants) that are common in pre-tests of questionaires may fail to uncover even common problems. A default sample size of 30 participants is recommended.

Profitability and resource use efficiency of maize seed production in Palpa district of Nepal

“…Pre-testing of questionnaire was done in Madanpokhara VDC with 10 respondents (5.5 percent of the sample size), which is common in pre-test of questionnaire (Perneger et al, 2015) and more relevant in our case constrained by the resources. The pre-tested questionnaire was administered to collect primary data in June, 2016.…”

Guidelines for developing, translating, and validating a questionnaire in perioperative and pain medicine

“…Before conducting a pilot test of the questionnaire on the intended respondents, it is advisable to test the questionnaire items on a small sample (about 30–50)[21] of respondents. [17] This is an opportunity for the questionnaire developer to know if there is confusion about any items, and whether respondents have suggestions for possible improvements of the items.…”

“…As with developing a new questionnaire, the prefinal version of the translated questionnaire should be pilot tested on a small sample (about 30–50)[21] of the intended respondents. [2526] After completing the translated questionnaire, the respondent is asked (verbally by an interviewer or via an open-ended question) to elaborate what they thought each questionnaire item and their corresponding response meant.…”

Are Wastewater Systems Adapting to Climate Change?

“…To improve measurement reliability and validity, the survey was pilot tested with nine individuals including wastewater managers not in the final survey sample, wastewater experts at the CTDEEP, and experts in questionnaire design in October 2015 (Schultz and Whitney ; Perneger et al. ). The survey was revised using pilot testing feedback and the revised survey was administered via email to wastewater managers at 131 wastewater systems in Connecticut from November 2015 to April 2016 (Qualtrics, Provo, Utah, USA) with three email reminders to encourage a response (Dillman et al.…”

“…), and the broader context (e.g., public or political support, regulatory environment) drawing on relevant theoretical literature on factors thought to drive adaptation actions including risk perceptions and organizational learning Lesnikowski et al 2011;Berkhout 2012). To improve measurement reliability and validity, the survey was pilot tested with nine individuals including wastewater managers not in the final survey sample, wastewater experts at the CTDEEP, and experts in questionnaire design in October 2015 (Schultz and Whitney 2005;Perneger et al 2015).…”