Using Student Focus Groups to Support the Validation of Rubrics for Large Scale Undergraduate Independent Research Projects

Abstract

Finding methods of validating rubrics forsignificant “capstone” experiences, including fourth yeardesign projects and the research-oriented thesis, can bechallenging, given the large number of individualstypically involved in the assessment of studentdeliverables. This paper describes a methodology forusing student focus groups to support the validation of arubric for a fourth year thesis course in a largeEngineering Program, and the results from these focusgroups. Through focus group discussion and activitysheets used in the focus groups, a number of interestinginsights were raised about both the rubric, namely: a lackof consultation by the students with the rubric until thefinal stages of writing the final report; concerns andinconsistencies in the perception of how supervisors willuse the rubric; a perceived lack of focus on process andproject experience-related criteria and concerns with thelevel of expectation of the project experience-relatedcriteria that are present, and other concerns related toterminology and distance between rubric descriptors. Thefocus group provided a useful forum for discussion oncourse experience and assessment, effectively allowingstudents to both individually reflect, and build on eachother’s ideas and suggestions.

Testing Inter-Rater Reliability in Rubrics for Large Scale Undergraduate Independent Projects

“…Twentyfour students participated, representing a variety of supervisors, program majors (and therefore thesis topics), as well as a variety of performance levels. The results of the focus group were described in [13], and students were also asked to self-assess using the rubric. These selfassessments are not discussed in this paper, due to time limitations, but will be the subject of a future paper.…”

“…In previous work, the student interpretation of the rubric requirements and the applicability of the rubrics to individual student projects, through the use of student focus groups, has been described [13]. This work elicited a better understanding of how the students use the rubric, vocabulary in the rubric that was unclear, and the students' impression of how their supervisor was (or perhaps, was not) using the grading rubric.…”

“…Previous papers have discussed the challenges involved in rubric development in this context, particularly those presented by the variety of projects and expectations of instructors involved in assessment of student deliverables, as well as the use of focus groups to gain a deeper understanding of student perspectives on course deliverables and rubric criteria [2,3,13]. This research has led to revisions in the rubrics deployed in the course which, we hypothesize, best balance the needs of all stakeholders in the process.…”

Engineering Instructor Use and Interpretation of Rubrics in a Large Scale Undergraduate Independent Research Course

“…In previous papers, we have described the challenges posed to universal rubric development by the variety and number of projects, as well as the number of supervisors, who have varying levels of experience managing an undergraduate capstone experience [1][2][3]8]. Attempts to validate the rubrics on the student side -through focus groups and interviews -have led to important insights into how they understand and employ the rubrics [8], revealing an anxiety about how supervisors use them (or don't use them) as assessment tools. These concerns lead us to test for inter-rater reliability, comparing the results of two "rubric expert" raters with the supervisors: we identified substantial agreement between the "rubric experts," but only fair agreement between those raters and actual project supervisors [3].…”

Work in Progress: Developing Single Point Rubrics for Formative Assessment

“…To address the prior review issue, students will be asked to examine each of the single point rubrics, first individually, and then collectively within their project groups, and answer the following questions: 16  What is unclear on the rubric?  What is not assessed in the rubric, but should be?…”