Stop Worrying about Multiple-Choice: Fact Knowledge Does Not Change with Response Format

Abstract

Declarative fact knowledge is a key component of crystallized intelligence. It is typically measured with multiple-choice (MC) items. Other response formats, such as open-ended formats are less frequently used, although these formats might be superior for measuring crystallized intelligence. Whereas MC formats presumably only require recognizing the correct response to a question, open-ended formats supposedly require cognitive processes such as searching for, retrieving, and actively deciding on a response from long-term memory. If the methods of inquiry alter the cognitive processes involved, mean-changes between methods for assessing declarative knowledge should come along with changes in the covariance structure. We tested these assumptions in two online studies administering declarative knowledge items in different response formats (MC, open-ended, and open-ended with cues). Item difficulty clearly increases in the open-ended methods although effects in logistic regression models vary slightly across items. Importantly, latent variable analyses suggest that the method of inquiry does not affect what is measured with different response formats. These findings clearly endorse the position that crystallized intelligence does not change as a function of the response format.

Driving factors of individual differences in broad retrieval ability: Gr is more than the sum of its parts.

“…We went beyond previous research regarding Gc and Gr, as our Gc factor was indicated through a broad range of vocabulary and declarative knowledge tests in different response formats (i.e., multiple-choice vs. open-ended). Further, the reported measurement model of Gc replicates earlier research showing that tests of vocabulary and declarative knowledge share a common core (Schipolowski et al, 2014) and that response format plays no role in individual differences in declarative knowledge (Goecke et al, 2022). The predictive power of Gc toward Gr implies that Gc explains incremental variance in Gr that is not accounted for by the other covariates.…”

“…Given vocabulary and fact knowledge measures of Gc are very strongly related (Schipolowski et al, 2014), their relation with Gr should be next to indistinguishable. Interestingly, comparing open-ended and closed fact knowledge response formats (i.e., varying the retrieval demands in Gc tasks) did not reveal meaningful differences in correlations in a recent study (Goecke et al, 2022), which underlines the idea that Gc and Gr should be related independent of task instantiations.…”

Public events knowledge in an age-heterogeneous sample: Reminiscence bump or bummer?

“…PUBLIC EVENTS KNOWLEDGE 13 with divergent biographies or cultural backgrounds (Watrin et al, 2023). Furthermore, the present study focuses on the recognition of public events using a multiple-choice response format rather than an open-ended one, which has been reported to be more agesensitive ; see also Goecke et al, 2022, for the structural consistency of closed and open-ended standardized knowledge tests). We deem it important to complement and further explain the current results by concurrently investigating closed and open-ended response formats to further explore a potential reminiscence bump.…”

Empirical Validation of a Learning Progression in Initial Chemistry Education

“…The assessment typically involves multiple-choice items (Emden et al, 2018;Neumann et al, 2013) or ordered multiple-choice items (Briggs et al, 2006;Todd et al, 2017). Chen et al (2016) and Goecke et al (2022) have discussed the influence of item formats. Goecke et al (2022) state that the method of inquiry does not affect what is measured with different response formats, while Chen et al (2016) opt for mixed formats.…”

“…Chen et al (2016) and Goecke et al (2022) have discussed the influence of item formats. Goecke et al (2022) state that the method of inquiry does not affect what is measured with different response formats, while Chen et al (2016) opt for mixed formats. A review by Harris et al (2022) shows that the majority of studies utilizes short-answer or fixed response tasks.…”