Exploring the Impact of Random Guessing in Distractor Analysis

Abstract

Multiple-choice (MC) items are widely used in educational tests. Distractor analysis, an important procedure for checking the utility of response options within an MC item, can be readily implemented in the framework of item response theory (IRT). Although random guessing is a popular behavior of test-takers when answering MC items, none of the existing IRT models for distractor analysis have considered the influence of random guessing in this process. In this article, we propose a new IRT model to distinguish the influence of random guessing from response option functioning. A brief simulation study was conducted to examine the parameter recovery of the proposed model. To demonstrate its effectiveness, the new model was applied to the mathematics tests of the Hong Kong Diploma of Secondary Education Examination (HKDSE) from 2015 to 2019. The results of empirical analyses suggest that the complexity of item contents is a key factor in inducing students' random guessing. The implications and applications of the new model to other testing situations are also discussed.Multiple-choice (MC) questions are undoubtedly the most popular format used in existing academic assessments. An MC item is composed of a stem and several (usually 2−5) response options (Lord, 1977), from which test-takers are expected to select the only correct option or the best option among the given ones. The incorrect options are also referred to as "distractors." The design of distractors is directly related to the difficulty of an MC item. A good MC item should have an accurate identification of the correct options and distractors (Fellenz, 2004). Moreover, distractors should be sufficiently attractive. A distractor that fails to attract test-takers should be revised or dropped (Kehoe, 1995), because an MC item without effective distractors would be less discriminating (Haladyna & Downing, 1993). Thus, distractor analysis is necessary to ensure the measurement quality of MC items.Many methods can be used to detect implausible distractors (Gierl et al., 2017). An intuitive strategy is to observe the frequency of options used by test-takers. Specifically, practitioners can easily flag nonfunctioning distractors that are infrequently (e.g., <5%) used in a test (Rao et al., 2016;Tarrant et al., 2009). Additionally, to examine the discrimination of distractors, researchers have also calculated the mean score of test-takers who choose a distractor (Haladyna & Rodriguez, 2013) or the point-biserial correlation for a distractor (Attali & Fraenkel, 2000). Alternatively,

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model with Novel Process Data Indicators

“…However, even though disengaged responding might be related to the ability of the test-taker or the item difficulty, disengaged responses do not reflect these properties in the same way as responses stemming from solution behavior. As a consequence, simply ignoring disengaged responding can lead to biased item and person parameter estimates (Jin et al, 2022, Rios et al, 2017. Furthermore, it can also influence conclusions and decisions on other aspects such as differential item functioning (DeMars & Wise, 2010) or the speed-ability relation (Deribo et al, 2022).…”

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model With Novel Process Data Indicators

“…However, even though disengaged responding might be related to the ability of the test-taker or the item difficulty, disengaged responses do not reflect these properties in the same way as responses stemming from solution behavior. As a consequence, simply ignoring disengaged responding can lead to biased item and person parameter estimates ( Jin et al, 2022 ; Rios et al, 2017 ). Furthermore, it can also influence conclusions and decisions on other aspects such as differential item functioning ( DeMars & Wise, 2010 ) or the speed-ability relation ( Deribo et al, 2022 ).…”

Perbandingan Keberfungsian Distraktor Dalam Tes: Pendekatan Teori Tes Klasik Dan Modern Dalam Konteks Pendidikan Sekolah Dasar

“…Sejalan dengan itu penelitian oleh maulana menunjukkan bahwa analisis data menunjukkan perlunya penyederhanaan rentang pilihan jawaban, juga variasi tingkat kesulitan butir yang tidak terlalu beragam sehingga sebagian besar responden tidak dapat terukur dengan baik (Maulana, Rangkuti, & Wahyuni, 2020). Dalam penelitian yang berbeda disebutkan bahwa analisis distraktor merupakan prosedur penting untuk memeriksa kegunaan opsi respons dalam soal pilihan ganda dan dapat dengan mudah diimplementasikan dalam kerangka teori respons butir (Teori tes modern), dan prilaku menebak opsi jawaban secara acak adalah perilaku populer peserta tes saat menjawab soal pilihan ganda sehngga perlu dibuat alternatif analisi supaya dapat membuat opsi pilihan ganda tidak mudah ditebak dan distraktor berfungsi maksimal (Jin & Siu, 2022). Terkait dengan distraktor dan daya beda menunjukkan bahwa efisiensi distraktor tidak berkorelasi dalam pola yang konsisten dengan indeks diskriminasi (daya beda), hanya 50% distraktor yang efisien melakukan pengukuran (Puthiaparampil & Rahman, 2021).…”