A Comparison of the Quality of Data-Driven Programming Hint Generation Algorithms

Price, Thomas W.; Dong, Yihuan; Zhi, Rui; Paaßen, Benjamin; Lytle, Nicholas; Cateté, Veronica; Barnes, Tiffany

doi:10.1007/s40593-019-00177-z

Cited by 37 publications

(14 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Supporting self-regulated learning skills, specifically planning support, could lead to successful problem-solving in computer programing courses (Chi et al, 2014; Schraw, 2007). Fortunately, researchers have focused on suggesting learner support systems for computer programing learning (e.g., Blikstein et al, 2014; Seufert, 2018; Price et al, 2019; Vrieling et al, 2018). We also suggest that these efforts could be more fruitful if we consider learners’ problem-solving skills, self-regulated learning skills, and cognitive load as shown in this study.…”

Section: Discussionmentioning

confidence: 99%

The Effects of Self-Regulated Learning Support on Learners’ Task Performance and Cognitive Load in Computer Programing

Shin

Song

2022

Journal of Educational Computing Research

View full text Add to dashboard Cite

This study explores the effect of self-regulated learning support on learners’ cognitive load and problem-solving performance, considering cases from well-structured to ill-structured tasks in a computer programing course. Sixty-seven undergraduate students in a computer programing fundamentals course were randomly assigned into one of two groups: (1) experimental group ( N = 34; received self-regulated learning support) and (2) control group ( N = 33). Participants in both groups were asked to solve programing problems using Python over 3 weeks, and their cognitive load levels were measured after each week. The results showed that novice learners’ self-regulated learning skills can be influenced by learning support, which contributes to solving complex problems. The results also suggest that, given the sequence of tasks (i.e., basic and advanced well-structured and ill-structured problems), self-regulated learning support may require time to be effective. The germane load of the experimental group was higher for well-structured problems than that of the control group. Using logistic regression, the two groups were differentiated through different variables measured in this study, which showed a high predictive explanatory power. The results provide implications for designing self-regulated learning support for programing courses, using problem-based learning in association with problem-solving skills, self-regulated learning skills, and cognitive load management.

show abstract

Section: Discussionmentioning

confidence: 99%

The Effects of Self-Regulated Learning Support on Learners’ Task Performance and Cognitive Load in Computer Programing

Shin

Song

2022

Journal of Educational Computing Research

View full text Add to dashboard Cite

show abstract

“…In addition, agreement or disagreement with experts does not necessarily imply high or low quality hints. These types of techniques were also compared in [17].…”

Section: Discussionmentioning

confidence: 99%

“…To address this issue, many studies have employed a range of different evaluation methods, including user studies and surveys [14,9,6,1,2], comparisons with experts [12,10], analysis using historical data [8,5,7,13,9,4,15,11,16,3] or other technical evaluations [15]. Researchers have also conducted comparison studies on small subsets of techniques [17,12]. However, the time-consuming nature of evaluations, complexity of hint techniques and the apparently disparate nature of hints produced by these techniques (e.g.…”

Section: Introductionmentioning

confidence: 99%

A Survey of Automated Programming Hint Generation -- The HINTS Framework

McBroom,

Koprinska,

Yacef

2019

Preprint

View full text Add to dashboard Cite

Automated tutoring systems offer the flexibility and scalability necessary to facilitate the provision of high quality and universally accessible programming education. In order to realise the full potential of these systems, recent work has proposed a diverse range of techniques for automatically generating hints to assist students with programming exercises. This paper integrates these apparently disparate approaches into a coherent whole. Specifically, it emphasises that all hint techniques can be understood as a series of simpler components with similar properties. Using this insight, it presents a simple framework for describing such techniques, the Hint Iteration by Narrow-down and Transformation Steps (HINTS) framework, and it surveys recent work in the context of this framework. It discusses important implications of the survey and framework, including the need to further develop evaluation methods and the importance of considering hint technique components when designing, communicating and evaluating hint systems. Ultimately, this paper is designed to facilitate future opportunities for the development, extension and comparison of automated programming hint techniques in order to maximise their educational potential.

show abstract

“…There is a growing body of research on automated hint generation for programming exercises (McBroom et al, 2019;Price et al, 2019). Most work in this area is concerned with detection or generation of a suitable sequence of hints to provide to students at specific points during their learning, and hints are mainly generated using templates combining mixed-language input (Rivers, 2017).…”

Section: Natural Language-based Interactions In Itsmentioning

confidence: 99%

Automated Data-Driven Generation of Personalized Pedagogical Interventions in Intelligent Tutoring Systems

Kochmar

Belfer³

et al. 2021

Int J Artif Intell Educ

View full text Add to dashboard Cite

Intelligent tutoring systems (ITS) have been shown to be highly effective at promoting learning as compared to other computer-based instructional approaches. However, many ITS rely heavily on expert design and hand-crafted rules. This makes them difficult to build and transfer across domains and limits their potential efficacy. In this paper, we investigate how feedback in a large-scale ITS can be automatically generated in a data-driven way, and more specifically how personalization of feedback can lead to improvements in student performance outcomes. First, in this paper we propose a machine learning approach to generate personalized feedback in an automated way, which takes individual needs of students into account, while alleviating the need of expert intervention and design of hand-crafted rules. We leverage state-of-the-art machine learning and natural language processing techniques to provide students with personalized feedback using hints and Wikipedia-based explanations. Second, we demonstrate that personalized feedback leads to improved success rates at solving exercises in practice: our personalized feedback model is used in , a large-scale dialogue-based ITS with around 20,000 students launched in 2019. We present the results of experiments with students and show that the automated, data-driven, personalized feedback leads to a significant overall improvement of 22.95% in student performance outcomes and substantial improvements in the subjective evaluation of the feedback.

show abstract

A Comparison of the Quality of Data-Driven Programming Hint Generation Algorithms

Cited by 37 publications

References 39 publications

The Effects of Self-Regulated Learning Support on Learners’ Task Performance and Cognitive Load in Computer Programing

The Effects of Self-Regulated Learning Support on Learners’ Task Performance and Cognitive Load in Computer Programing

A Survey of Automated Programming Hint Generation -- The HINTS Framework

Automated Data-Driven Generation of Personalized Pedagogical Interventions in Intelligent Tutoring Systems

Contact Info

Product

Resources

About