Response Options and the Measurement of Political Knowledge

Abstract

By many measures, the public knows little about politics. But just how little people seem to know depends on the questions that are put to them. In particular, knowledge levels seem higher when people are asked closed- rather than open-ended questions. In turn, differences between estimated knowledge levels are sometimes attributed to fundamental differences between these types of questions. Building on this previous research, the present study uses a pre-registered experiment conducted with a representative national sample to shed new light on the relationship between question form and knowledge measurement. The authors find that inferences about political knowledge depend less on fundamental differences between open- and closed-ended questions than on two little-appreciated aspects of survey design: the number and difficulty of the response options that accompany closed-ended questions. These aspects of survey design have large effects. Scholars who use the same questions with different response options may reach substantively different conclusions about the public's levels of knowledge.

How long does it take to admit that you do not know? Gender differences in response time to political knowledge questions

“…To maximize the quality of the political knowledge scale and discourage cheating, we implemented four measures: (1) A short introduction was presented to the respondents reminding them that there were not being graded and any answer was acceptable; (2) a time limit was imposed in each of the questions to 30 or 40 s depending on the length of the wording (Clifford and Jerit 2016). Once the time ran out, the next question appeared in the screen and respondents could not return to the preceding question; (3) different formats (namely, closed-ended questions with different number of categories, pictures, and click and drag questions) have been employed throughout the questionnaire to avoid respondents’ fatigue 3 and limit potential response biases and cheating related to question format (Bullock and Rader, 2021); and (4) the items had varying levels of difficulty (percentage of correct responses ranges from 7.8% to 88.8%).…”

Power-sharing and Identity-Politics Transformation in Zanzibar, Tanzania

“…This cast (73%) respondents out of 393 eligible for the study. The percentage is enough to undergo this study (Bullock & Rader, 2021;John, 2020;Mugenda & Mugenda, 2003). The study also consists of five key informants from Members of the House of Representatives, Power-sharing initiators and elites, citizens and political activists.…”

Women Also Know Stuff: Challenging the Gender Gap in Political Sophistication

“…olitical sophistication is a foundational concept in the study of political attitudes and behaviora crucial moderator impacting a range of mechanisms such as political decision-making and vote choice (Lau and Redlawsk 2001;Macdonald, Rabinowitz, and Listhaug 1995), persuasion and motivated reasoning (Lodge and Taber 2013;Zaller 1992), or the susceptibility to misinformation (Vegetti and Mancosu 2020). Yet fundamental concerns regarding the measurement of political sophistication continue to plague the discipline (Bullock and Rader 2022;Mondak 2001;Sturgis, Allum, and Smith 2008). Scholars usually rely on survey questions that assess people's ability to recall basic facts about political institutions and officeholders as a proxy for sophistication (Barabas et al 2014;Delli Carpini and Keeter 1993).…”