A Recommender System Based on Effort: Towards Minimising Negative Affects and Maximising Achievement in CS1 Learning

Abstract

Programming online judges (POJs) are autograders that have been increasingly used in introductory programming courses (also known as CS1) since these systems provide instantaneous and accurate feedback for learners' codes solutions and reduce instructors' workload in evaluating the assignments. Nonetheless, learners typically struggle to find problems in POJs that are adequate for their programming skills. A potential reason is that POJs present problems with varied categories and difficulty levels, which may cause a cognitive overload, due to the large amount of information (and choice) presented to the student. Thus, students can often feel less capable, which may result in undesirable affective states, such as frustration and demotivation, decreasing their performance and potentially leading to increasing dropout rates. Recently, new research emerged on systems to recommend problems in POJs; however, the data collection for these approaches was not fine-grained; importantly, they did not take into consideration the students' previous effort and achievement. Thus, this study proposes for the first time a prescriptive analytics solution for students' programming behaviour by constructing and evaluating an automatic recommender module based on students' effort, to personalise the problems presented to the learner in POJs. The aim is to improve the learners achievement, whilst minimising negative affective states in CS1 courses. Results in a within-subject double-blind controlled experiment showed that our method significantly improved positive affective states, whilst minimising the negatives ones. Moreover, our recommender significantly increased students' achievement (correct solutions) and reduced dropout and failure in problem-solving.

Towards Supporting Cs1 Instructors and Learners With Fine-Grained Topic Detection in Online Judges

“…Importantly, OJs provide many problems for students and instructors [54,56,20,40]. The further can use the system for to develop their skills in specific computer science topics such as graphs, dynamic programming, and in beginning concepts, such as vectors, matrices and so forth.…”

“…Thus, it is important to propose strategies to automate the organisation and annotation of OJ questions with useful meta-information. Indeed, automatic classification methods to detect the problems' categories in OJ are essential for instructors and students of computer science majors [54,25,40]. Moreover, a potential automation in the annotation would bring many benefits to OJ administrators, since, to the best of our knowledge, the annotation is performed manually in the few OJs in which problems are labelled with topics.…”

“…Indeed, in recent works [27,40] about recommending problems in OJs to CS1 students and instructors, the authors explained that a recommender could be useful for students, by allowing them to select appropriate problems unassisted, and for instructors, by allowing them to select problems to compose lists of exercises and tests. Nonetheless, the recommender method has a limitation related to problem topics.…”

“…Online Judge (OJ) systems are a special kind of MOOC that provides a reliable, automatic, and instantaneous evaluation of an algorithm's source code sent by the learners [54,25,42,50,26,39,36]. The popularity of OJ systems are increasing in CS1 classes [54,20,37,11,57,49], due to their conveniences for learners (automatic code correction) and instructors (workload reduction in the students' code correction) [51,8,12,23,40,41,10]. Indeed, OJs facilitate the use of the Many Small Programs (MSP) approach [4,3,44,28], which requires the students to solve many smaller programming assignments weekly or biweekly, instead of a traditional One Larger Program (OLP), in which the learners solve a unique larger assignment in a short period of time.…”

“…Despite the benefits of OJs for programming learning, recent studies [54,37,20,14,41] report that OJ systems need to be further improved to serve students, instructors, and administrators better. In this sense, one of the biggest issues of these systems is that the available problems are not organised based on meta-information that facilitates the users' searching process [56,25,40,54,41]. More specifically, problems are usually arranged in volumes, without any classification based on topics.…”