Question-by-Question Interrater Analysis and Suggestions for Improvements of a Functional Model Scoring Rubric

Abstract

Functional modeling is a tool used for system abstraction. By divorcing system function from component structure, functional modeling allows designers to more easily identify design opportunities and compartmentalize product functions, which can lead to innovation during the ideation process. In this paper, we examine the reliability of a rubric used to evaluate student-generated functional models by comparing interrater reliabilities on a question-by-question basis from a previous study where an examination of the reliability of each question was not assessed. We then suggest changes to the rubric in order to improve the rubric’s overall interrater reliability as well as its question-by-question interrater reliability. These rubric alterations include clarification of vague language, inclusion of examples and counter examples, and a procedure for handling nonexistent functional components as opposed to incorrect or “nonsensical” functional components. This work is in contribution to the ongoing development of this functional modeling rubric as an education instrument. As functional modeling becomes more widely accepted in the design community and in engineering curricula, it is important to have a validated evaluation metric with which to assess student-generated functional models.

Validation of a Mental Model Elicitation Instrument through Deployment of Control Groups in an Undergraduate Engineering Program

“…The research presented in this paper fits into a larger frame of research on students' ability to understand systems. Recently, this research includes analogical reasoning [2,5] and the impact of functional modeling and decomposition training on systems representation [1,3], while historically, efforts were focused on methods for teaching function to undergraduate students [1,[31][32][33][45][46][47][48][49][50][51]. The more recent inclusion of studies including graduate students [1] and nonengineering students (in progress) are helping the authors explore different ways to bring technology literacy into both engineering and non-engineering classrooms.…”

“…The previously published mental model scoring rubrics used in this study build off of previously published work on functional modeling and decomposition [30][31][32], and in particular the final 20-question functional modeling rubric [33]. For example, a portion of the mental model rubrics (High-Level Systems Thinking) use similar language to the functional modeling rubric to intentionally measure students' functional understanding of the system, which is an important aspect of a mental model considering the definition provided by Fein et al [8].…”

Board 18: The Impact of Functional Modeling in an Engineering Design Curriculum

“…The previously discussed rubric ambiguity [6] was addressed by the PIs in 2018 [10] by calculating the interrater agreement at the question level and making suggestions for improvement. One such solution to the issue of ambiguity was quantifying the parameters of the rubric such as stating "at least 80%" instead of "overwhelming majority".…”

“…A summary of research efforts thus far and their related theme(s) is summarized in Table 1. [1][2][3][4][5][6][7][8][9][10][11][12][13] In the 2011 paper surveying functional modeling teaching methods [1] two sub questions were identified to help determine the value functional modeling brings to the design process: 1.) How do you quantitatively assess the quality of any given functional model?…”

“…The initial scoring rubric was successful with relatively high inter-rater reliability [4,5], but also allowed for some interpretation discrepancies [6]. After initial studies using the scaffold approach to teaching and the binary rubric for functional model assessment, additional work sought to assess the impact of teaching function [8,9] as well as rubric refinement [10,12]. Results thus far have indicated that teaching function provides students a mechanism to decompose engineered systems [9,13], and preliminary data suggests the recognition of function improves students' mental models [11].…”