Utilizing programming traces to explore and model the dimensions of novices' code‐writing skill

Abstract

Studies have found that most novice programmers have low proficiency in writing code. However, it is unclear what subskills compose code writing and which subskills novice programmers struggle with. This study utilizes programming traces to identify latent subskills that constitute code writing so that teachers can offer specific instruction on the weak subskills. Data were collected from an undergraduate course teaching introductory computer science in Java. Six hundred and fourteen students made submissions to homework programming questions in a web‐based learning system. Based on the submission traces, we computed 11 features related to correctness and time students spent on their submissions. We conducted an exploratory factor analysis on two‐thirds of students selected randomly and identified four factors. The first factor, code style proficiency, was mainly related to code style errors. The second, syntactic proficiency, concerned compiler errors. The third is semantic proficiency, which concerns runtime and logic errors. The fourth, syntactic debugging proficiency, concerned the success rate and time required for fixing compiler and code style errors. A confirmatory factor analysis conducted on the remaining one‐third of the data supported the four‐factor structure. The factor model showed measurement invariance between the data set where the model was developed and two new datasets, one from the same sample but collected at a different time point and another from a different sample and context (onsite course vs. online course). The factors were related to prior programming abilities, programming language familiarity, and future exam performance. These associations provided validity evidence for the factor model.

Investigating the reliability of aggregate measurements of learning process data: From theory to practice

“…Research has used the proportion of submissions to programming assignments containing syntax errors as a measurement of syntax proficiency (Zhang et al, 2023). We adopted this operationalization but acknowledged that this measurement might be just a facet of syntactic proficiency.…”

“…The current study used four dimensions of programming proficiency to illustrate segmenting event sequences to compute the reliability of aggregate measurements: programming syntactic, programming semantic, syntactic error debugging, and semantic error debugging proficiency. Research has used the proportion of submissions to programming assignments containing syntax errors as a measurement of syntax proficiency (Zhang et al, 2023). We adopted this operationalization but acknowledged that this measurement might be just a facet of syntactic proficiency.…”

Effect of Block-Based Python Programming Environment on Programming Learning

“…Furthermore, six constructs (psychological motivation, peer collaboration, cognitive problem solving, interaction with instructors, community support, and learning management) were extracted to conduct the study [34]. To determine why beginner programmers have low coding skill levels, factor analysis was conducted on the answers to programming problems submitted by 614 university students through a web-based learning system, and four skill-level constructs were extracted: code style, syntactic, logical error-related, and syntax debugging [35]. Factor analysis is a method used to extract underlying constructs from a set of observed variables, categorizing confirmed constructs based on shared variance and extracting content commonly explained by multiple evaluation questions.…”