Challenge of Analysis of Polytomous Item Characteristics with Item Response Theory

Abstract

In the world of education, a lot of scoring is done with a polytomous, for example, on items that are constructed responses. Likewise, at the Open University of Indonesia, the questions for the final exam of a course called the take-home exam (THE) are presented in the form of a constructed response. This problem is done by students who take this course, but the number of students who take this course is not stable. Sometimes, this course is taken by a few participants. On the one hand, it is necessary to carry out an analysis of item characteristics. On the other hand, doing so will face many challenges. In this study, the challenge of analyzing polytomous data is presented on the polytomous score in 3 courses whose final exam is presented in the form of a take-home exam (THE). This study is a mixed research, quantitative analysis packaged in qualitative research with a narrative tradition. Documentary data in the form of students' answers to 3 courses' take-home exams are then analyzed for their characteristics by using various models of IRT polytomous data analysis. Obstacles in conducting analysis are told in narrative form. The analysis was carried out on the take-home exam package of three subjects, namely the Statistical Method II course (89 test participants and two questions), Experimental Design (67 test participants and three questions), and the Sampling method (206 test participants and three questions). Based on the results of the analysis, there is only one package of questions that can be thoroughly analyzed with item response theory, namely the package of questions with the sampling method. Based on the analysis process, it was found that there are challenges in conducting an analysis with item response theory. The challenges are mastery of the R language, the syntax of the selected analysis package, the length or many items in one test package, many test takers, and, last, foresight in rescoring to produce a more proportional pattern. This limitation can be used as a consideration for other researchers in analyzing the polytomous data.