A Validated Scoring Rubric for Explain-in-Plain-English Questions

Chen, Binglin; Azad, Sushmita; Haldar, Rajarshi; West, Matthew; Zilles, Craig

doi:10.1145/3328778.3366879

Cited by 21 publications

(7 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We then significantly modified the rubric as we reviewed and discussed the writing samples. The coders played a significant role in further developing the rubric, which also helped improve their awareness of the writing issues they encountered.This is in line with prior work on rubric development [3,4,7,11,15,16,18,21].…”

Section: Rubric Design and Emergent Codingsupporting

confidence: 89%

Exploring Common Writing Issues in Upper-Year Computer Science

Munir

Strafforello

Kani

et al. 2022

Proceedings of the 53rd ACM Technical Symposium on Computer Science Education

View full text Add to dashboard Cite

This study analyzes common issues in the writing of our upperyear, undergraduate computer science students in timed (e.g. tests) and untimed (e.g. longer assignment reports) scenarios. Our goal is to identify writing issues that should be addressed earlier in the CS curriculum. In collaboration with a writing specialist, we develop and fine-tune a rubric with Grammar, Conciseness, Clarity, Organization, Structure, and Formality as the main categories.We find, in our writing samples, that grammatical issues (such as punctuation errors or run-on sentences) are common even in a setting where students have ample time to proofread. Other common issues include unclear pronoun antecedents, lacking topic sentences, and other structural and organizational issues. Moreover, we correlate the presence of these issues with the students' grades. We do not find statistically significant correlations between writing skill and grades, and the weak correlations we find depend heavily on the writing context.We hope that our rubric items and findings are useful for instructors in targeting writing skills that warrant development.

show abstract

Section: Rubric Design and Emergent Codingsupporting

confidence: 89%

Exploring Common Writing Issues in Upper-Year Computer Science

Munir

Strafforello

Kani

et al. 2022

Proceedings of the 53rd ACM Technical Symposium on Computer Science Education

View full text Add to dashboard Cite

show abstract

“…Several studies on assessing the internal consistency of rubrics in programming and other topics exist [2,3]. Nathalia et al confirmed the CodeMaster rubric using 88,812 app inventor projects [2].…”

Section: Evaluating the Reliability Of Rubricsmentioning

confidence: 99%

“…Nathalia et al confirmed the CodeMaster rubric using 88,812 app inventor projects [2]. Chen, et al [3] used the rubric to evaluate code reading and assessed consistency. Cronbach's alpha coefficient [7] was used to validate the rubrics in these studies.…”

Section: Evaluating the Reliability Of Rubricsmentioning

confidence: 99%

See 1 more Smart Citation

Validation of Rubric Evaluation for Programming Education

et al. 2021

View full text Add to dashboard Cite

In evaluating the learning achievement of programming-thinking skills, the method of using a rubric that describes evaluation items and evaluation stages is widely employed. However, few studies have evaluated the reliability, validity, and consistency of the rubrics themselves. In this study, we introduced a statistical method for evaluating the characteristics of rubrics using the goal question metric (GQM) method. Furthermore, we proposed a method for measuring four evaluation results and characteristics obtained from rubrics developed using this statistical method. Moreover, we showed and confirmed the consistency and validity of the statistical method using the GQM method of the resulting developed rubrics. We show how to verify the consistency and validity of the rubric using the GQM method.

show abstract

“…However, such a proposal immediately arrives at the question, "But, which answers should the algorithm mark as correct?" While we endeavored in prior work to establish a rubric for EipE [5], our experience using EipE produced myriad questions and difficult scoring decisions. In this work, we sought out to explore if there was a community consensus on how EipE questions should be graded.…”

Section: Introductionmentioning

confidence: 99%

How should we ‘Explain in plain English’? Voices from the Community

Fowler

Chen

Zilles

2021

Proceedings of the 17th ACM Conference on International Computing Education Research

Self Cite

View full text Add to dashboard Cite

Explain in plain English" (EipE) questions are seen as an important developmental activity and assessment tool in the research community studying how people learn to program, but they aren't widely used in practice because of difficulty of grading and workload issues. In this paper, we interviewed eleven members of the introductory programming education research community about their thoughts on EipE questions as a whole and how individual borderline student answers should be graded. Through inductive coding of the interview transcripts, we identify: (1) themes relating to how EipE questions should be used in class, (2) the importance of training students to complete EipE questions, (3) standards for the selection and presentation of code in EipE questions, (4) the theoretical and practical considerations relating to grading EipE questions, and (5) English as a second language (ESL) concerns. In addition, we attempt to extrapolate from our observations what the underlying grading process is that faculty are using to grade EipE questions. CCS CONCEPTS• Social and professional topics → Computing education.

show abstract

A Validated Scoring Rubric for Explain-in-Plain-English Questions

Cited by 21 publications

References 22 publications

Exploring Common Writing Issues in Upper-Year Computer Science

Exploring Common Writing Issues in Upper-Year Computer Science

Validation of Rubric Evaluation for Programming Education

How should we ‘Explain in plain English’? Voices from the Community

Contact Info

Product

Resources

About