Survey Pretesting: Do Different Methods Produce Different Results?

Presser, Stanley; Blair, Johnny

doi:10.2307/270979

Cited by 186 publications

(151 citation statements)

References 11 publications

Supporting

Mentioning

138

Contrasting

Unclassified

Order By: Relevance

“…Although many varieties of expert reviews exist , two primary goals of an expert review are to reveal problems with a survey instrument so that they can be remedied prior to going into the field or to sort items into groups that are more or less likely to exhibit measurement errors. For example, expert reviewers have been used as a pretesting method (Presser and Blair 1994;Esposito and Rothgeb 1997;, to identify problematic linguistic structures in survey questions (Holbrook et al 2007), and for classification of items related to interviewer effects (Mangione et al 1992;Schnell and Kreuter 2005). The number of expert reviewers tends to be small, ranging from two or three expert methodologists (Presser and Blair 1994;Theis et al 2002;Jansen and Hak 2005;Holbrook et al 2007) to over 20 reviewers .…”

Section: Introductionmentioning

confidence: 99%

“…For example, expert reviewers have been used as a pretesting method (Presser and Blair 1994;Esposito and Rothgeb 1997;, to identify problematic linguistic structures in survey questions (Holbrook et al 2007), and for classification of items related to interviewer effects (Mangione et al 1992;Schnell and Kreuter 2005). The number of expert reviewers tends to be small, ranging from two or three expert methodologists (Presser and Blair 1994;Theis et al 2002;Jansen and Hak 2005;Holbrook et al 2007) to over 20 reviewers . In general, expert review is a relatively quick and inexpensive method for evaluating questionnaires (Presser and Blair 1994).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An Examination of Questionnaire Evaluation by Expert Reviewers

2010

View full text Add to dashboard Cite

Expert reviews are frequently used as a questionnaire evaluation method but have received little empirical attention. Questions from two surveys are evaluated by six expert reviewers using a standardized evaluation form. Each of the questions has validation data available from records. Large inconsistencies in ratings across the six experts are found. Despite the lack of reliability, the average expert ratings successfully identify questions that had higher item nonresponse rates and higher levels of inaccurate reporting. This article provides empirical evidence that experts are able to discern questions that manifest data quality problems, even if individual experts vary in what they rate as being problematic. Compared to a publicly available computerized question evaluation tool, ratings by the human experts positively predict questions with data quality problems, whereas the computerized tool varies in success in identifying these questions. These results indicate that expert reviews have value in identifying question problems that result in lower survey data quality. 295

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

An Examination of Questionnaire Evaluation by Expert Reviewers

2010

View full text Add to dashboard Cite

show abstract

“…It is very important that the survey instruments are subject to adequate pretesting before the implementation of the survey (Presser and Blair 1994). Testing should be first done within the team, and then with the enumerators, and then pilot testing under field conditions by the enumerators.…”

Section: Test Test and Test Againmentioning

confidence: 99%

Impact Evaluation of Development Interventions:

2018

View full text Add to dashboard Cite

“…For each item that was perceived as problematic, we reviewed the transcripts of the interviews and coded them into problem types, using a problem classification scheme adopted from various existing schemes (DeMaio & Landreth, 2004;Lessler & Forsyth, 1996;Presser & Blair, 1994;Rothgeb, Willis, & Forsyth, 2001). The problem classification scheme included a total of 30 problem codes that were grouped according to the four stages of the survey response process (comprehension, retrieval, judgment, and response selection; Tourangeau, 1984;Tourangeau, Rips, & Rasinski, 2000;see Appendix B).…”

Section: Types Of Problemsmentioning

confidence: 99%

A Comparison of Two Cognitive Pretesting Techniques Supported by Eye Tracking

Neuert

Lenzner

2016

Social Science Computer Review

View full text Add to dashboard Cite

Nutzungsbedingungen:Dieser Text wird unter einer Deposit-Lizenz (Keine Weiterverbreitung -keine Bearbeitung) AbstractIn questionnaire pretesting, supplementing cognitive interviewing with eye tracking is a promising new method that provides additional insights into respondents' cognitive processes while answering survey questions. When incorporating eye tracking into cognitive interviewing, two retrospective probing techniques seem to be particularly useful. In the first technique -retrospective probing-participants complete an online questionnaire, while cognitive interviewers monitor participants' eye movements in an adjacent room and note down any peculiarities in their reading patterns. Afterward, the interviewers ask targeted probing questions about these peculiarities in a subsequent cognitive interview. In the second technique-gaze video cued retrospective probing-respondents are additionally shown a video of their eye movements during the cognitive interview. This video stimulus is supposed to serve as a visual cue that may better enable respondents to remember their thoughts while answering the questions. We examine whether one of the two techniques is more effective when it comes to identifying problematic survey questions. In a lab experiment, participants' eye movements (n = 42) were tracked while they completed six questions of an online questionnaire. Simultaneously, their reading patterns were monitored by an interviewer for evidence of response problems. After completion of the online survey, a cognitive interview was conducted. In the retrospective probing condition, probing questions were asked if peculiar reading patterns were observed during the eye-tracking session (e.g., rereadings of specific words or text passages). In the other condition, participants were shown a video of their recorded eye movements, in addition to receiving probing questions about the questions displayed. Results show that both techniques did not differ in terms of the total number of problems identified. However, gaze video cued retrospective probing identified fewer unique problems and fewer types of problems than pure retrospective probing.

show abstract

Survey Pretesting: Do Different Methods Produce Different Results?

Cited by 186 publications

References 11 publications

An Examination of Questionnaire Evaluation by Expert Reviewers

An Examination of Questionnaire Evaluation by Expert Reviewers

Impact Evaluation of Development Interventions:

A Comparison of Two Cognitive Pretesting Techniques Supported by Eye Tracking

Contact Info

Product

Resources

About