E- Assessment with Multiple-Choice Questions: A 5 Year Study of Students’ Opinions and Experience

Abstract

Aim/Purpose: The aim of this study is to understand student’s opinions and perceptions about e-assessment when the assessment process was changed from the traditional computer assisted method to a multiple-choice Moodle based method. Background: In order to implement continuous assessment to a large number of students, several shifts are necessary, which implies as many different tests as the number of shifts required. Consequently, it is difficult to ensure homogeneity through the different tests and a huge amount of grading time is needed. These problems related to the traditional assessment based on computer assisted tests, lead to a re-design of the assessment resulting in the use of multiple-choice Moodle tests. Methodology: A longitudinal, concurrent, mixed method study was implemented over a five-year period. A survey was developed and carried out by 815 undergraduate students who experienced the electronic multiple-choice questions (eMCQ) assessment in the courses of the IS department. Qualitative analyses included open-ended survey responses and interviews with repeating students in the first year. Contribution: This study provides a reflection tool on how to incorporate frequent moments of assessment in courses with a high number of students without overloading teachers with a huge workload. The research analysed the efficiency of assessing non-theoretical topics using eMCQ, while ensuring the homogeneity of assessment tests, which needs to be complemented with other assessment methods in order to assure that students develop and acquire the expected skills and competencies. Findings: The students involved in the study appreciate the online multiple-choice quiz assessment method and perceive it as fair but have a contradictory opinion regarding the preference of the assessment method, throughout the years. These changes in perception may be related to the improvement of the question bank and categorisation of questions according to difficulty level, which lead to the nullification of the ‘luck factor’. Other major findings are that although the online multiple-choice quizzes are used with success in the assessment of theoretical topics, the same is not in evidence regarding practical topics. Therefore, this assessment needs to be complemented with other methods in order to achieve the expected learning outcomes. Recommendations for Practitioners: In order to be able to evaluate the same expected learning outcomes in practical topics, particularly in technology and information systems subjects, the evaluator should complement the online multiple-choice quiz assessment with other approaches, such as a PBL method, homework assignments, and/or other tasks performed during the semester. Recommendation for Researchers: This study explores e-assessment with online multiple-choice quizzes in higher education. It provides a survey that can be applied in other institutions that are also using online multiple-choice quizzes to assess non-theorical topics. In order to better understand the students’ opinions on the development of skills and competencies with online multiple-choice quizzes and on the other hand with classical computer assisted assessment, it would be necessary to add questions concerning these aspects. It would then be interesting to compare the findings of this study with the results from other institutions. Impact on Society: The increasing number of students in higher education has led to a raised use of e-assessment activities, since it can provide a fast and efficient manner to assess a high number of students. Therefore, this research provides meaningful insight of the stakeholders’ perceptions of online multiple-choice quizzes about practical topics. Future Research: An interesting study, in the future, would be to obtain the opinions of a particular set of students on two tests, one of the tests using online multiple-choice quizzes and the other through a classical computer assisted assessment method. A natural extension of the present study is a comparative analysis regarding the grades obtained by students who performed one or another type of assessment (online multiple-choice quizzes vs. classical computer assisted assessment).

A systematic review on trends in using Moodle for teaching and learning

“…Nevertheless, most papers focused on summative assessment based on Moodle quizzes investigating both teachers’ and students’ opinions when implementing multiple-choice questions (Babo & Suhonen, 2018 ; Cakiroglu et al, 2017 ; Dimic et al, 2018 ; McKenzie & Roodenburg, 2017 ; Shdaifat & Obeidallah, 2019 ). According to a 5-year study, the ‘luck’ factor associated with multiple-choice questions is fair (Babo et al, 2020 ). Studies that have investigated the students' point of view indicate that the students agree that Moodle is easy to use and complements teaching, although most students still prefer classical assessment techniques (Cakiroglu et al, 2017 ; McVey, 2016 ; Popovic et al, 2018 ).…”

The Challenges of Online Teaching in COVID-19 Pandemic: A Case Study of Public Universities in Karachi, Pakistan

“…Accordingly, this study suggests that both faculty members and students should be helped with training sessions to get familiar with the online assessment techniques. Furthermore, online assessments should be harmonized with other techniques to accomplish the projected learning objectives (Babo et al, 2020).…”

Online Assessment in Higher Education: A Systematic Review

“…Concerning the design of formative online assessment through quizzes, the key publications suggest that quizzes should not only include true or false questions but a mixture of types (López-Tocón, 2021). Other studies found that quizzes are a well-suited form of online assessment for theoretical knowledge, but not necessarily for practical knowledge and should therefore be combined with other forms of online assessment, such as project-based learning or further homework tasks (Babo et al, 2020). While a higher correlation between final exam performance and the performance on formative quizzes with limited time and attempts was found (MacKenzie, 2019), unlimited attempts in general lead to a higher performance in the final exam (Davis et al, 2020).…”

“…Implementation type: Automated assessment was mostly used on quizzes. Quizzes included all sorts of tasks that did not require students to write longer answer, such as multiplechoice questions, single-choice questions, blank-filling or crossword-type tests (Admiraal et al, 2015;Azevedo et al, 2022;Babo et al, 2020;Bacca-Acosta & Avila-Garzon, 2021;Bekmanova et al, 2021;Carpenter et al, 2017;Chaudy & Connolly, 2018;Davis et al, 2020;Dermo & Boyne, 2014;Förster et al, 2018;Gamage et al, 2019;Gámiz Sánchez et al, 2014: Guerrero-Roldán & Noguera, 2018Hughes et al, 2020;Huisman et al, 2018;Kühbeck et al, 2019;López-Tocón, 2021;Mao & Peck, 2013;Meek et al, 2017;Mora et al, 2012;Ortega-Arranz et al, 2019;Reilly et al, 2016;Ross et al, 2018;Sancho-Vinuesa et al, 2018;Schaffer et al, 2017;Shaw et al, 2019;Stratling, 2017: Taghizadeh et al, 2014Tempelaar, 2020;Thille et al, 2014;Wilkinson et al, 2020). Automated assessment, including Natural Language Processing, was also used on short-answer questions (Carnegie, 2015;Chen et al, 2018;Ellis & Barber, 2016), or tasks including longer texts (Reilly et al, 2016;Santamaría Lancho et al, 2018;Xian, 2020).…”

“…In 23 cases, an automated assessment was used in the context of solely formative assessment (Acosta-Gonzaga & Walet, 2018; Bacca-Acosta & Avila-Garzon, 2021; Bekmanova et al, 2021;Carpenter et al, 2017;Chen et al, 2021;Förster et al, 2018;Gámiz Sánchez et al, 2014;Hughes et al, 2020;Kühbeck et al, 2019;López-Tocón, 2021;MacKenzie, 2019;Meek et al, 2017;Ogange et al, 2018;Polito & Temperini, 2021;Reilly et al, 2016;Sancho-Vinuesa et al, 2018;Santamaría Lancho et al, 2018;Scalise et al, 2018;Schaffer et al, 2017;Stratling, 2017;Tempelaar, 2020;Wilkinson et al, 2020;Xian, 2020). A total of 15 cases included formative as well as summative assessments (Admiraal et al, 2015;Azevedo et al, 2022;Babo et al, 2020;Carnegie, 2015;Davis et al, 2020;Dermo & Boyne, 2014;Gamage et al, 2019;Guerrero-Roldán & Noguera, 2018;Huisman et al, 2018;Mora et al, 2012;Nguyen et al, 2017;Ortega-Arranz et al, 2019, Wells et al, 2021Yang et al, 2016, Turner & Briggs, 2018 and only six cases used automated assessment exclusively as summative assessment (Chaudy & Connolly, 2018;…”