ProgSnap2: A Flexible Format for Programming Process Data

Price, Thomas W.; Hovemeyer, David; Rivers, Kelly; Gao, Ge; Bart, Austin Cory; Kazerouni, Ayaan M.; Becker, Brett A.; Petersen, Andrew; Gusukuma, Luke; Edwards, Stephen H.; Babcock, David

doi:10.1145/3341525.3387373

Cited by 19 publications

(9 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hence, the quantity and quality of the data they produce increases signiicantly, enabling the extraction of unprecedented knowledge about students, such as their behavioral patterns [36,51] and learning laws [59]. In this sense, eforts to standardize the collection, sharing, and analysis of such data are underway [193]. Notwithstanding, there is still a large amount of wasted data (e.g., even though most of the surveyed tools provide an embedded editor, only ive capture code changes, of which only two capture at keystroke granularity).…”

Section: Discussionmentioning

confidence: 99%

“…A wider overview of the body of knowledge regarding the use of learning analytics for the learning of programming has been performed by the Working Group Report of the 2015 ITiCSE [113]. Recently, a standardized format for logging programming process data -ProgSnap2 -arose from the requirements, experiences, and datasets of a large working group of researchers in the ield of computing education [193]. This work addresses one of the most signiicant gaps in the learning analytics of automated assessment data, which is the lack of a standard format to collect, share, and analyze data produced in automated assessment tools.…”

Section: Learning Analytics (Rq5)mentioning

confidence: 99%

“…Following the recent eforts to standardize data produced from automated assessment [193], works adhering to the proposed format and even extending it are expected. For instance, adapting the format to store subsequent code states of the same program as changes in the AST (or other intermediate representation of source-code) or reasoning on previous collected data to suggest ixes to new students facing diiculties.…”

Section: Future Trends and Research Directions (Rq7)mentioning

confidence: 99%

See 2 more Smart Citations

Automated Assessment in Computer Science Education: A State-of-the-Art Review

Paiva

Leal

Figueira

2022

ACM Trans. Comput. Educ.

125

View full text Add to dashboard Cite

Practical programming competencies are critical to the success in computer science education and go-to-market of fresh graduates. Acquiring the required level of skills is a long journey of discovery, trial and error, and optimization seeking through a broad range of programming activities that learners must perform themselves. It is not reasonable to consider that teachers could evaluate all attempts that the average learner should develop multiplied by the number of students enrolled in a course, much less in a timely, deeply, and fairly fashion. Unsurprisingly, exploring the formal structure of programs to automate the assessment of certain features has long been a hot topic among CS education practitioners. Assessing a program is considerably more complex than asserting its functional correctness, as the proliferation of tools and techniques in the literature over the past decades indicates. Program efficiency, behavior, readability, among many other features, assessed either statically or dynamically, are now also relevant for automatic evaluation. The outcome of an evaluation evolved from the primordial boolean values to information about errors and tips on how to advance, possibly taking into account similar solutions. This work surveys the state-of-the-art in the automated assessment of CS assignments, focusing on the supported types of exercises, security measures adopted, testing techniques used, type of feedback produced, and the information they offer the teacher to understand and optimize learning. A new era of automated assessment, capitalizing on static analysis techniques and containerization, has been identified. Furthermore, this review presents several other findings from the conducted review, discusses the current challenges of the field, and proposes some future research directions.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Learning Analytics (Rq5)mentioning

confidence: 99%

Section: Future Trends and Research Directions (Rq7)mentioning

confidence: 99%

See 1 more Smart Citation

Automated Assessment in Computer Science Education: A State-of-the-Art Review

Paiva

Leal

Figueira

2022

ACM Trans. Comput. Educ.

125

View full text Add to dashboard Cite

show abstract

“…They additionally also consider aspects of LA and find that only 3 out of 30 assessment tools offer more than simple statistics for the teachers; those three tools provide in particular insights into the program development process of the learners (Paiva et al, 2022, Table 7). The authors furthermore conclude that a significant gap in this field is the lack of a standardised data format; this has partly been addressed by the proposal of Price et al (2020).…”

Section: Assessments In Educationmentioning

confidence: 99%

Formative assessment strategies for students' conceptions—The potential of learning analytics

Stanja

Gritz

Krugel

et al. 2022

Brit J Educational Tech

View full text Add to dashboard Cite

Formative assessment is considered to be helpful in students' learning support and teaching design. Following Aufschnaiter's and Alonzo's framework, formative assessment practices of teachers can be subdivided into three practices: eliciting evidence, interpreting evidence and responding. Since students' conceptions are judged to be important for meaningful learning across disciplines, teachers are required to assess their students' conceptions. The focus of this article lies on the discussion of learning analytics for supporting the assessment of students' conceptions in class. The existing and potential contributions of learning analytics are discussed related to the named formative assessment framework in order to enhance the teachers' options to consider individual students' conceptions. We refer to findings from biology and computer science education on existing assessment tools and identify limitations and potentials with respect to the assessment of students' conceptions.

show abstract

“…A major part of compilation-related research aims to identify common errors and the time required to fix them or the evolution of error activity over time [1,3,10,11,22,31], as well as common fixes that students use to correct their programs [14]. Other work aims to identify "at-risk" students by looking at their programming patterns [8,13,17], apply programming efficiency measures to investigate the behavior of users in different regions [20,32], error message content and presentation [21,23], provide categorization of error messages [29], improve them [7], automate the identification and correction of syntax errors [33], and developing standards for programming process data [30].…”

Section: Prior Workmentioning

confidence: 99%

Sympathy for the (Novice) Developer

Karvelas

Becker

2022

Proceedings of the 53rd ACM Technical Symposium on Computer Science Education

Self Cite

View full text Add to dashboard Cite

In this work we investigate compilation behavior and error resolution time of thousands of novice programmers using two different versions of the BlueJ pedagogical Java programming environment. The two versions feature different compilation and error message presentation mechanisms. BlueJ 3 is a click-to-compile environment with enforced first error message presentation whereas BlueJ 4 features automatic compilation and on-demand error message presentation. We provide an overview of compilation activity based on novices' programming time spent in different compilation states and also present differences in the time it takes students to reach successful compilation states in each version. We find that compilation activity differs for each version and that students reach successful compilations faster in BlueJ 4. Based on our findings, we discuss the effectiveness of each version and argue against the constant red underlining of code -an error indicator mechanism present in BlueJ 4 -as this can lead to false positive errors that may mislead novices during their programming tasks. CCS CONCEPTS• Social and professional topics → Computing education; CS1.

show abstract

ProgSnap2: A Flexible Format for Programming Process Data

Abstract: Some rights reserved. For more information, please see the item record link above.

Cited by 19 publications

References 24 publications

Automated Assessment in Computer Science Education: A State-of-the-Art Review

Automated Assessment in Computer Science Education: A State-of-the-Art Review

Formative assessment strategies for students' conceptions—The potential of learning analytics

Sympathy for the (Novice) Developer

Contact Info

Product

Resources

About