A Function-Based Scoring Method for Evaluating Student Mental Models of Systems

Abstract

Mental models are loosely-defined constructs people form to reason and make predictions about their surroundings. These models are an important aspect of systems thinking for engineers, a concept that emphasizes holistic thinking when working with complex systems which is increasingly important in multiple engineering disciplines. Methods to evaluate systems thinking and mental models of systems traditionally rely on questionnaires, or detailed interactive simulations of specific processes. This work presents a method based on functional modules for evaluating student responses to an instrument based on Lawson’s bicycle problem, intended to elicit students’ mental models of two systems. Students were given a simple outline of the two systems, a hair dryer and a car radiator, and were prompted to fill and label the components required for the system to fulfil the functionality described. This was done in two sessions, once before learning functional modeling, and once after, to utilize the method of scoring to evaluate any changes in their mental models due to exposure to functional modeling. The scoring method identifies common functional modules between two systems using Module Heuristics, and then identifies students’ recognition of those modules. This allows a direct comparison of the functional similarity between the two systems identified by the students and can capture a wider variety of correct answers than simply counting the components a student provides.

Validation of a Mental Model Elicitation Instrument through Deployment of Control Groups in an Undergraduate Engineering Program

“…Taken as a whole, the results suggest that practice effects are not present when implementing the mental model instruments, which answers the original question that motivated the study. This helps to validate the instrument for future studies, as well as validate the authors' previously published work [1][2][3]5]. From a broad perspective, this project aims to understand the role of function in systems understanding, students' ability to communicate their knowledge about systems through mental model representations, and to find avenues for bringing technological literacy into both engineering and non-engineering classrooms.…”

“…During the initial phases of the project, students in engineering design courses were given a series of experimental instruments or homework assignments to assess their ability to recognize product functionality, interpret and understand customer needs, and to explain or decompose a complex system. Students in prior studies had either previously learned functional modeling [3] or were taught functional modeling as an intervention between different mental model instrument implementations [1,2,5]. The investigators slowly adapted these instruments to assess students' systems thinking aptitude and have recently begun to investigate the relationship between mental models, recognition of function, and technology literacy and understanding.…”

Exploring Differences in Senior and Sophomore Engineering Students’ Mental Models of Common Products

“…All participants completed the exercise during the allotted time of 30 minutes. The data was then analyzed, comparing the students' responses to an expert-generated solution such as the ones generated in [2]. This was done by assessing whether the student response matched the expert response for each component given.…”

“…An abbreviated version of the HD rubric is provided in the appendix as Table A.1. The solutions were also evaluated on a function basis in which all the ways a student satisfied a function were noted and were scored on criteria adapted from a previously published and revised functional modeling rubric [2], [5], [17] respectively. The mental model instruments were scored on several criteria as either 0, 0.5, or 1 based on the level of completeness.…”

“…Consider the following skills being requested of new engineering graduates to be prepared to join the 21 st century engineering workforce: adaptability, complex communication and social skills, self-management and self-development, and systems thinking [1]. For engineers, mental models serve a crucial role in systems thinking [2] as the cognitive interpretations of the universe and the organization, perception, and structure of knowledge [3]. Mental models allow one to be adaptable and to self-manage as they provide a framework for estimating the systems and systems of systems that engineers often find themselves working within.…”