Multiple-true-false questions reveal more thoroughly the complexity of student thinking than multiple-choice questions: a Bayesian item response model comparison

Abstract

Background: Within undergraduate science courses, instructors often assess student thinking using closed-ended question formats, such as multiple-choice (MC) and multiple-true-false (MTF), where students provide answers with respect to predetermined response options. While MC and MTF questions both consist of a question stem followed by a series of options, MC questions require students to select just one answer, whereas MTF questions enable students to evaluate each option as either true or false. We employed an experimental design in which identical questions were posed to students in either format and used Bayesian item response modeling to understand how responses in each format compared to inferred student thinking regarding the different options. Results: Our data support a quantitative model in which students approach each question with varying degrees of comprehension, which we label as mastery, partial mastery, and informed reasoning, rather than uniform random guessing. MTF responses more closely estimate the proportion of students inferred to have complete mastery of all the answer options as well as more accurately identify students holding misconceptions. The depth of instructional information elicited by MTF questions is demonstrated by the ability of MTF results to predict the MC results, but not vice-versa. We further discuss how MTF responses can be processed and interpreted by instructors. Conclusions: This research supports the hypothesis that students approach MC and MTF questions with varying levels of understanding and demonstrates that the MTF format has a greater capacity to characterize student thinking regarding the various response options.

Interactive learning modules with 3D printed models improve student understanding of protein structure–function relationships

“…We designed our assessments to measure learning of the targeted objectives. The assessments included multiple-choice and multiple-true-false questions, which required students to evaluate multiple statements describing a single scenario and helped us to diagnose misconceptions [20][21][22]. We scored multiple-choice questions with an equal weight to individual multiple-true-false statements.…”

Teaching Metabolism in Upper-Division Undergraduate Biochemistry Courses using Online Computational Systems and Dynamical Models Improves Student Performance

“…, 2017 ; Couch et al. , 2018 ; Brassil and Couch, 2019 ). Each question in our assessments was unique, and successive assessments were not designed to evaluate longitudinal learning of a concept (e.g., the “energy charge” concept was measured in the Glycolysis, TCA, and ETC assessments, but the questions were unique to each system).…”

“…We used multiple true-false (MTF) questions consisting of a question stem that is presented together with a series of statements that students evaluate as true or false (Supporting Files S4 and S8). We selected MTF questions because they can reveal student misconceptions that remain undetected in free-response and multiple-choice question formats (Hubbard et al, 2017;Couch et al, 2018;Brassil and Couch, 2019). Each question in our assessments was unique, and successive assessments were not designed to evaluate longitudinal learning of a concept (e.g., the "energy charge" concept was measured in the Glycolysis, TCA, and ETC assessments, but the questions were unique to each system).…”

Using computational modeling to teach metabolism as a dynamic system improves student performance

“…For our assessments, we used multiple-true-false (MTF) questions that consist of a question stem that is presented together with a series of statements that are evaluated by students as being true or false. We selected MTF questions because they can reveal student misconceptions that remain undetected in freeresponse and multiple-choice question formats (53)(54)(55). The number of statements for each question is listed in Tables 1 and 2. Before class, students were asked to complete a Cell Collective training module to familiarize themselves with the technology and modeling concepts and to complete the pre-assessment questions online.…”