Systems for Automated Evaluation of Programming Tasks: Development, Use and Perspectives

Abstract

The paper is devoted to the issues related to the development and use of automated assessment systems for programming tasks. The work on automated testing of programming tasks had been held in Novosibirsk State University since 1998. It was created several generations of systems of this type. The first program of about 1,000 lines was written in Perl. Currently, the automated system, named NSUts, is used in the programming contests and in the educational process. As a result of work in it, as well as acquaintance with other similar systems, both Russian and foreign, it was possible to formulate requirements for software development of this kind, to analyze the difference between the systems for testing competitions and training support systems. The article also provides an overview of the existing systems for checking tasks on programming, an analysis of the advantages and disadvantages of each of these systems, and a detailed comparison of the NSUts system with the Yandex Contest system. The current state of the NSUts system architecture is described. Much attention is paid to the practice of using the system in the educational process, statistics and graphs of students' progress of students studying with the help of this system are given. The article describes the analysis of the experience of using the NSUts system and outlines the prospects for its development.

Development of an automated system for conducting, checking and evaluating programming competitions

“…Under this model, the software is not installed on users' computers, but on the provider's servers, and the user does not have any access to the system. This disadvantage makes the system unsuitable for longterm contests due to the fact that the terms of the service provider can change at any moment and the user (in our case, the organizer of the contest) will not be able to do anything [2]. One of the most successful systems in history was PCSM2, which was formed by combining the APPES and PCMS systems and is characterized by the flexibility of the internal structure.…”