How shall we assess this?

Carter, Janet; Ala-Mutka, Kirsti; Fuller, Ursula; Dick, Martin; English, John; Fone, William; Sheard, Judy

doi:10.1145/960492.960539

Cited by 85 publications

(21 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Contact Publisher@InformingScience.org to request redistribution permission. Ala-Mutka (2005) provides a critique of a number of automated assessment initiatives; Carter et al report on the results of an international teaching staff survey of automated assessment (Carter et al, 2003), and Rosenthal (2004) showed a clear preference for automated assessment in blind tests. Students can benefit from instant feedback, consistent and impartial marking, and the ability to progress at their own pace.…”

Section: Introductionmentioning

confidence: 99%

“…Many people still believe that 'automated assessment' and 'multiple choice tests' are effectively synonymous (Carter et al, 2003), and even when shown how freetext questions can be marked, refuse to believe that they can assess the ineffable qualities they look for when marking by hand, in the same way as many people used to believe that computers could never play chess as well as a human being because of a lack of strategic vision and human intuition. In the light of the experiences we have described here and the extremely positive feedback we have received, we hope that this paper will help to convince others to try to use automated assessment in their own courses.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Experiences of Using Automated Assessment in Computer Science Courses

English¹,

English²

2015

JITE:IIP

Self Cite

View full text Add to dashboard Cite

In this paper we discuss the use of automated assessment in a variety of computer science courses that have been taught at Israel Academic College by the authors. The course assignments were assessed entirely automatically using Checkpoint, a web-based automated assessment framework. The assignments all used free-text questions (where the students type in their own answers). Students were allowed to correct errors based on feedback provided by the system and resubmit their answers. A total of 141 students were surveyed to assess their opinions of this approach, and we analysed their responses. Analysis of the questionnaire showed a low correlation between questions, indicating the statistical independence of the individual questions. As a whole, student feedback on using Checkpoint was very positive, emphasizing the benefits of multiple attempts, impartial marking, and a quick turnaround time for submissions. Many students said that Checkpoint gave them confidence in learning and motivation to practise. Students also said that the detailed feedback that Checkpoint generated when their programs failed helped them understand their mistakes and how to correct them.

show abstract

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

Experiences of Using Automated Assessment in Computer Science Courses

English¹,

English²

2015

JITE:IIP

Self Cite

View full text Add to dashboard Cite

show abstract

“…In his earlier work, Ercole et al (2002) studied collusion in a comparative empirical study using multiple choice questions in face-to-face and online examinations. It is one of the major security threats (Laubscher et al 2005) which challenges the validity of online examinations (Carter et al 2003). It can be classified in the following categories based on its occurrence in different scenarios (Ullah et al 2016…”

Section: Background and Related Work 21 Academic Dishonesty And Collmentioning

confidence: 99%

A study into the usability and security implications of text and image based challenge questions in the context of online examination

2018

View full text Add to dashboard Cite

Online examinations are an integral component of online learning environments and research studies have identified academic dishonesty as a critical threat to the credibility of such examinations. Academic dishonesty exists in many forms. Collusion is seen as a major security threat, wherein a student invites a third party for help or to impersonate him or her in an online examination. This work aims to investigate the authentication of students using text-based and image-based challenge questions. The study reported in this paper involved 70 online participants from nine countries completing a five week online course and simulating an abuse case scenario. The results of a usability analysis suggested that i) image-based questions are more usable than text-based questions (p < 0.01) and ii) using a more flexible data entry method increased the usability of text-based questions (p < 0.01). An impersonation abuse scenario was simulated to test the influence of sharing with different database sizes. The findings revealed that iii) an increase in the number of questions shared for impersonation increased the success of an impersonation attack and the results showed a significant linear trend (p < 0.01). However, the number of correct answers decreased when the attacker had to memorize and answer the questions in an invigilated online examination or their response to questions was timed. The study also revealed that iv) Educ Inf Technol (2019) an increase in the size of challenge question database decreased the success of an impersonation attack (p < 0.01).

show abstract

“…First, students often collaborate with peers and teaching assistants while working on the assignments. While we encourage students to ask questions and collaborate, it is difficult point out which part of the code a student has figured out himself and which part he or she could not have completed without help from the peers or teachers (Carter et al, 2003). For these reasons, an exam, an open-book exam in our case, can be seen as a reasonably reliable way to assess a CS exam.…”

Section: Introductionmentioning

confidence: 99%

“…Besides these benefits from the student's point of view, practical programming in computer-based environment provides teachers a way of evaluating students' application and problem-solving skills in a semi-or fully automatic fashion (computer-based or computer aided assessment, see, Carter et al, 2003;Russell et al, 2003). Yet, final exams in CS1 are often conducted with penand-paper.…”

Section: Introductionmentioning

confidence: 99%

Towards Computer-based Exams in CS1

Lappalainen

Lakanen

Högmander

2017

Proceedings of the 9th International Conference on Computer Supported Education

View full text Add to dashboard Cite

Abstract:Even though IDEs are often a central tool when learning to program in CS1, many teachers still lean on paperbased exams. In this study, we examine the "test mode effect" in CS1 exams using the Rainfall problem. The test mode was two-phased. Half of the participants started working on the problem with pen and paper, while the other half had access to an IDE. After submitting their solution, all students could rework their solution on an IDE. The experiment was repeated twice during subsequent course instances. The results were mixed. From the marking perspective, there was no statistically significant difference resulting from the mode. However, the students starting with the paper-based part tended to make more errors in their code, but after the computer-based reworking phase, they matched or exceeded the level of the students who started with the computer-based phase. We also discuss the reliability of automatic assessment that is based on a unit test suite that was developed for the purposes of this study.

show abstract

How shall we assess this?

Cited by 85 publications

References 48 publications

Experiences of Using Automated Assessment in Computer Science Courses

Experiences of Using Automated Assessment in Computer Science Courses

A study into the usability and security implications of text and image based challenge questions in the context of online examination

Towards Computer-based Exams in CS1

Contact Info

Product

Resources

About