Using Learning Analytics to evaluate the quality of multiple-choice questions

Abstract

Purpose The purpose of this paper is to find appropriate forms of analysis of multiple-choice questions (MCQ) to obtain an assessment method, as fair as possible, for the students. The authors intend to ascertain if it is possible to control the quality of the MCQ contained in a bank of questions, implemented in Moodle, presenting some evidence with Item Response Theory (IRT) and Classical Test Theory (CTT). The used techniques can be considered a type of Descriptive Learning Analytics since they allow the measurement, collection, analysis and reporting of data generated from students’ assessment. Design/methodology/approach A representative data set of students’ grades from tests, randomly generated with a bank of questions implemented in Moodle, was used for analysis. The data were extracted from a Moodle database using MySQL with an ODBC connector, and collected in MS ExcelTM worksheets, and appropriate macros programmed with VBA were used. The analysis with the CTT was done through appropriate MS ExcelTM formulas, and the analysis with the IRT was approached with an MS ExcelTM add-in. Findings The Difficulty and Discrimination Indexes were calculated for all the questions having enough answers. It was found that the majority of the questions presented values for these indexes, which leads to a conclusion that they have quality. The analysis also showed that the bank of questions presents some internal consistency and, consequently, some reliability. Groups of questions with similar features were obtained, which is very important for the teacher to develop tests as fair as possible. Originality/value The main contribution and originality that can be found in this research is the definition of groups of questions with similar features, regarding their difficulty and discrimination properties. These groups allow the identification of difficulty levels in the questions on the bank of questions, thus allowing teachers to build tests, randomly generated with Moodle, that include questions with several difficulty levels in the tests, as it should be done. As far as the authors’ knowledge, there are no similar results in the literature.

Examining the Psychometric Properties of the Career Commitment Instrument through Classical Test Theory and the Graded Response Model

“…Measurement of standard error and index of difference d are in the appropriate category. Traditional test theory was chosen in analysing instrument quality because it is simpler and easier to understand (Azevedo et al, 2019;Foster, 2020). In addition to its understanding, which does not necessitate in-depth knowledge of the statistical distribution function and its mathematical models, this theory has a high practical value in explaining reliability and validity problems.…”

Extraction, Processing and Visualization of Peer Assessment Data in Moodle

“…When planning to extract data or extend the functionality of a Moodle module, such as in this case, there are three design alternatives ( Table 2). The first design approach for data extraction is to craft a tailored SQL (structured query language) query to the database that returns a given dataset based on the parameters used in the query; this option requires either the use direct access to the database and the use of additional drivers (such as in [57]) or the use of custom reporting plug-ins (e.g., https://docs.moodle.org/37/en/Custom_SQL_queries_report), as well as the necessary privileges, knowledge about SQL syntax, and the structure of Moodle database. Furthermore, it is highly dependent on the specific internal identifiers of the course and workshop, which requires adaptation whenever a change affecting these identifiers occurs.…”

Psychometric properties career of commitment instrument using classical test theory and graded response model

“…The pure score theory is another name for the classical test theory. It is related to the focus of classical test theory research, which seeks to see the pure score of the visible score obtained (Azevedo et al, 2019). Traditional test theory is simpler and easier to grasp.…”