Fifteen distractor two-level multiple choice items were developed as diagnostic instruments to evaluate the level of conceptual understanding and structure of students' misconceptions in explaining redox reactions. Questions at the first tier (Q1) assess the level of knowledge, and questions at the second tier (Q2) assess the level of reasoning of students. This instrument was given to 1150 participants. The participants were 11th grade students, from eight senior high schools, in the Eastern part of Indonesia. The collected data was analyzed using the Rasch model approach. The results of this study provide diagnostic and summative information on the progressiveness of student learning outcomes, as well as evidence of empirical validity and reliability of measurement. In addition, by comparing the size of items Q1 with Q2, it was found that the level of student knowledge is not always proportional to the level of reasoning, even in some cases, the level of knowledge is lower than the level of reasoning, and vice versa. The results of the investigation using the option probability curve; it was revealed that there were students' misconceptions and inconsistencies about the concepts of reduction, oxidation and oxidation numbers. This result confirms why students have difficulty interpreting and converting redox reaction equations.