We present a simple technique for evaluating multiple-choice questions and their answers beyond the usual measures of difficulty and the effectiveness of distractors. The technique involves the construction and qualitative consideration of item response curves and is based on item response theory from the field of education measurement. To demonstrate the technique, we apply item response curve analysis to three questions from the Force Concept Inventory. Item response curve analysis allows us to characterize qualitatively whether these questions are efficient, where efficient is defined in terms of the construction, performance, and discrimination of a question and its answer choices. This technique can be used to develop future multiple-choice examination questions and a better understanding of results from existing diagnostic instruments.

Several years ago, we introduced the idea of item response curves (IRC), a simplistic form of item response theory (IRT), to the physics education research community as a way to examine item performance on diagnostic instruments such as the Force Concept Inventory (FCI). We noted that a full-blown analysis using IRT would be a next logical step, which several authors have since taken. In this paper, we show that our simple approach not only yields similar conclusions in the analysis of the performance of items on the FCI to the more sophisticated and complex IRT analyses but also permits additional insights by characterizing both the correct and incorrect answer choices. Our IRC approach can be applied to a variety of multiple-choice assessments but, as applied to a carefully designed instrument such as the FCI, allows us to probe student understanding as a function of ability level through an examination of each answer choice. We imagine that physics teachers could use IRC analysis to identify prominent misconceptions and tailor their instruction to combat those misconceptions, fulfilling the FCI authors’ original intentions for its use. Furthermore, the IRC analysis can assist test designers to improve their assessments by identifying nonfunctioning distractors that can be replaced with distractors attractive to students at various ability levels.

No abstract

In the last few years, flipped teaching has become one of the most popular teaching trends being adopted. Yet, there is a lack of data on what students use and find useful while taking these classes. This paper focuses on the use of class web logs, student surveys and class records to analyze student use of the material developed as well as how this use correlates with course results. The course being investigated is taught once a year by the same professor. It is the second course in an introductory physics course sequence for life science majors. The flipped format was adopted gradually in the course starting the Spring 2009 semester. The analysis focuses on the last iteration. The current format of the course was partially driven by student input, and took several years to fully implement. In this case, lecture time is devoted to answer and discuss questions, work on practice problems, and, sometimes, to explore the topics students find interesting. Instead of lectures students complete online multimedia quizzes, embedding both short lecture type recording segments simulations and videos. The quiz format is meant to help students identify the topics they don't understand. Homework is also completed online. It includes both traditional end of the chapter problems and simulation mediated questions. Students also complete pre-laboratory simulation mediated activities. The analysis focused on the analysis of each of the components the students complete online. That data was also correlated to the student performance in various class assignments.