Dataset of Program Source Codes Solving Unique Programming Exercises Generated by Digital Teaching Assistant

Abstract

This paper presents a dataset containing automatically collected source codes solving unique programming exercises of different types. The programming exercises were automatically generated by the Digital Teaching Assistant (DTA) system that automates a massive Python programming course at MIREA—Russian Technological University (RTU MIREA). Source codes of the small programs grouped by the type of the solved task can be used for benchmarking source code classification and clustering algorithms. Moreover, the data can be used for training intelligent program synthesizers or benchmarking mutation testing frameworks, and more applications are yet to be discovered. We describe the architecture of the DTA system, aiming to provide detailed insight regarding how and why the dataset was collected. In addition, we describe the algorithms responsible for source code analysis in the DTA system. These algorithms use vector representations of programs based on Markov chains, compute pairwise Jensen–Shannon divergences of programs, and apply hierarchical clustering algorithms in order to automatically discover high-level concepts used by students while solving unique tasks. The proposed approach can be incorporated into massive programming courses when there is a need to identify approaches implemented by students.

A Rule-Based Algorithm and Its Specializations for Measuring the Complexity of Software in Educational Digital Environments

“…In addition, we calculated the occurrence frequencies of different code complexity metrics for each task type, and the results are shown in Figure 9. As is shown in Figure 10, the EduC metric is the strictest metric, rejecting 1603 more programs when compared to the CycC metric, which was previously used in the DTA system [25,45]. We suggest the use of the same threshold for EduC as for CycC (EduC ≤ 10) in autograding systems automating educational programming courses, including the DTA system.…”

“…The DTA system supports automatic generation and automatic checking of tasks of 11 types, including formal notation into source code translation tasks and conversion between data format tasks. The task types supported in DTA are [45]:…”

“…Examples of each of the listed task types are available in [45], and examples of programs solving unique programming exercises of 11 types are also available in [45]. However, as can be seen from the source codes of the accepted programs, the system accepts code that is poorly readable despite the presence of the CycC check.…”

“…In the Spring semester of 2023, the DTA system received 100,691 programs [45] solving unique automatically generated programming exercises. A total of 18,683 programs were automatically checked and accepted by the system, and 82,008 programs were rejected.…”

“…Histograms of the occurrence frequencies of the scaled CycC, CogC, and EduC metric values are shown in Figure 8 with logarithmic scaling. The metric values were obtained from programs from the dataset [45] (see Figure 7). The set X containing normalized values of a metric was computed as:…”