Modeling Guessing Components in the Measurement of Political Knowledge

Abstract

Due to the crucial role of political knowledge in democratic participation, the measurement of political knowledge has been a major concern in the discipline of political science. Common formats used for political knowledge questions include multiple-choice items and open-ended identification questions. The conventional wisdom holds that multiple-choice items induce guessing behavior, which leads to underestimated item-difficulty parameters and biased estimates of political knowledge. This article examines guessing behavior in multiple-choice items and argues that a successful guess requires certain levels of knowledge conditional on the difficulties of items. To deal with this issue, we propose a Bayesian IRT guessing model that accommodates the guessing components of item responses. The proposed model is applied to analyzing survey data in Taiwan, and the results show that the proposed model appropriately describes the guessing components based on respondents’ levels of political knowledge and item characteristics. That is, in general, partially informed respondents are more likely to have a successful guess because well-informed respondents do not need to guess and barely informed ones are highly seducible by the attractive distractors. We also examine the gender gap in political knowledge and find that, even when the guessing effect is accounted for, men are more knowledgeable than women about political affairs, which is consistent with the literature.

What do voters know, and why does it matter? Investigating issue‐specific knowledge and candidate choice in the 2020 U.S. primaries

“…(2014), our five questions covered temporal (static and surveillance) and topical (policy‐specific and general) dimensions. To avoid problems with differential rates of “don't know” responses, we used a forced‐choice, five‐option multiple choice format, which can more accurately assess partially informed participants (Tsai & Lin, 2017).…”

“…To avoid problems with differential rates of "don't know" responses, we used a forced-choice, fiveoption multiple choice format, which can more accurately assess partially informed participants (Tsai & Lin, 2017).…”

Modeling Partial Knowledge in Multiple-Choice Cognitive Diagnostic Assessment

“…The literature has revealed that examinees frequently exclude some options from the possible answers based on their partial knowledge before guessing (e.g., Lindquist & Hoover, 2015). This behavior is referred to as educated guessing (Wu et al, 2021) or informed guessing (Tsai & Lin, 2017;Wu et al, 2019). Fitting the MC-DINA or MC-S-DINA models to the responses associated with partial knowledge would lead to the overestimation of attribute mastery states because the models that do not consider partial knowledge tend to under-evaluate the probability of correctly answering the items.…”

When “Don’t Know” Indicates Nonignorable Missingness: Using the Estimation of Political Knowledge as an Example

“… 3. Some research discusses guessing behavior and differentiates between blind guessing and informed guessing. Informed guessing is arguably more common than blind guessing when answering multiple-choice questions (Delli Carpini and Keeter, 1996; Mehrens and Lehmann, 1987; Mondak, 1999); it has been shown that partially informed respondents are more likely to guess the correct answers than uninformed or misinformed ones (Tsai and Lin, 2017). …”