Measuring Political Knowledge and Not Search Proficiency in Online Surveys

Abstract

Most online survey questions testing political knowledge are susceptible to measurement error when participants look up the answers. This article reports five studies of methods to detect and prevent this common source of error. To detect lookups, “catch questions” are more reliable than self-reports, because many participants lie rather than admit looking up answers. Strongly worded instructions reduced lookups by about two-thirds, while the triple combination of instructions, requesting a promise not to look up answers, and adaptive feedback (asking participants who look up an answer to stop doing so) reduced the percentage of respondents looking up an answer by a further half, to 3%. For office recall knowledge items, photo-based open-ended questions eliminated lookups and had similar validity to traditional text-based versions, making them a good choice when a visual format is viable.