Online judges are systems designed for the reliable evaluation of algorithm
source code submitted by users, which is next compiled and tested in a
homogeneous environment. Online judges are becoming popular in various
applications. Thus, we would like to review the state of the art for these
systems. We classify them according to their principal objectives into systems
supporting organization of competitive programming contests, enhancing
education and recruitment processes, facilitating the solving of data mining
challenges, online compilers and development platforms integrated as components
of other custom systems. Moreover, we introduce a formal definition of an
online judge system and summarize the common evaluation methodology supported
by such systems. Finally, we briefly discuss an Optil.io platform as an example
of an online judge system, which has been proposed for the solving of complex
optimization problems. We also analyze the competition results conducted using
this platform. The competition proved that online judge systems, strengthened
by crowdsourcing concepts, can be successfully applied to accurately and
efficiently solve complex industrial- and science-driven challenges.Comment: Authors pre-print of the article accepted for publication in ACM
Computing Surveys (accepted on 19-Sep-2017