QUAID: A questionnaire evaluation aid for survey methodologists

Abstract

and KENTMARQUIS United States Census Bureau, Washington, D.C.QUAID (question-understanding aid) is a software tool that assists survey methodologists, social scientists, and designers of questionnaires in improving the wording, syntax, and semantics of questions. The tool identifies potential problems that respondents might have in comprehending the meaning of questions on questionnaires. These problems can be scrutinized by researchers when they revise questions to improve question comprehension and, thereby, enhance the reliability and validity of answers. QUAID was designed to identify nine classes of problems, but only five of these problems are addressed in this article: unfamiliar technical term, vague or imprecise relative term, vague or ambiguous noun phrase, complex syntax, and working memory overload. Wecompared the output of QUAID with ratings of language experts who evaluated a corpus of questions on the five classes of problems. The corpus consisted of 505 questions on 11 surveys developed by the U.S. Census Bureau. Analyses of hit rates, false alarm rates, d' scores, recall scores, and precision scores revealed that QUAID was able to identify these five problems with questions, although improvements in QUAID's performance are anticipated in future research and development.A good surveyor questionnaire contains questions that elicit valid and reliable answers from respondents in a short amount oftime. One of the challenges to survey researchers and social scientists is to design questions that achieve these general objectives. Researchers in the field ofsurvey methodology have proposed models that dissect the many stages of question answering (Cannell, Miller,

Interest, inferences, and learning from texts

“…It is important to have questions that are understandable because questions that are not understood are more likely to be answered inaccurately than questions that are understood (Graesser, Cai, Louwerse, & Daniel, 2006). The increase in inaccurate answers for questions that are difficult to understand compared to questions that are easy to understand may lead to lower reliability for test measures (Graesser, Wiemer-Hastings, Kreuz, Wiemer-Hastings, & Marquis, 2000). QUAID identifies nine categories of problems that may lead to questions that are difficult to understand: unfamiliar technical term, vague or imprecise predicate or relative term, vague or ambiguous noun phrase, complex syntax, working memory overload, misleading or incorrect presupposition, unclear question category, amalgamation of more than one question category, unclear question purpose, mismatch between question category and answer option, difficult to access specific of generic knowledge, and respondent unlikely to know answer (Graesser et al, 2006).…”

Comprehension of self-report evidence-based measures of anxiety

“…The QUAID was able to identify linguistic problems comparable to expert ratings. [12] The five core problems assessed are described below.…”

“…The Question Understanding Aid (QUAID) is a computer-based program, designed to identify and improve upon potential problems that may arise in the wording, syntax, and semantics of questions. [12] It was developed as a tool for aiding self-report measure development by identifying problematic linguistic elements. The QUAID was able to identify linguistic problems comparable to expert ratings.…”

Coh-Metrix: Analysis of text on cohesion and language

“…One important contribution of Coh-Metrix is that all of these modules are located in one central Web facility. Some of these modules incorporate or expand the modules that were developed in a Web facility that analyzes questions on surveys (called Question Understanding Aid, or QUAID; Graesser, K. Wiemer-Hastings, Kreuz, P. Wiemer-Hastings, & Marquis, 2000) and in a computer tutor that helps students learn about subject matter by holding conversations in natural language (called AutoTutor; Graesser, Person, Harter, & the Tutoring Research Group, 2001;Graesser, VanLehn, Rose, Jordan, & Harter, 2001).…”