Assessing problem-solving in science and engineering programs

Abstract

Science and engineering (S & E) programs frequently claim that they teach undergraduate students how to be good problem solvers. However, there has been little research to-date that demonstrates this, in no small part due to the fact that measuring problem-solving is quite difficult. Our recent work characterized experts' problemsolving as carrying out a series of several dozen decisions made in solving novel problems; these decisions are remarkably consistent across S & E disciplines. Based on this, we developed a template for an assessment that measures students' problem-solving skills by posing questions that require them to make a subset of these expert decisions in suitable contexts. Preliminary results show that the assessment captures a wide range of problem-solving skills among students, and exposes key weaknesses in problem-solving. Most importantly, the data show that students' predictive frameworks-their mental models of a system's key features and the relationships between them-are less robust than experts', limiting students' ability to make predictions and explain observations while problem-solving. We provide detailed results from a pilot-test of this assessment in the context of chemical engineering design, which applies many concepts from physics. These results point to a general deficiency in undergraduate S & E programs: students are not being given the opportunity to practice expert decision-making, and thus do not develop robust predictive frameworks by the end of their undergraduate programs.

Template for teaching and assessment of problem solving in introductory physics

“…An important area of future research is how we can effectively teach our students these expert predictive frameworks. Indeed, some preliminary investigations suggest that standard instructional practices and curriculum are not teaching these frameworks and their application to students in chemical engineering [40] or medicine [39], and our results here show that the same is true in physics courses at Stanford. Because predictive frameworks are mental models of problems' key features and the relationships between them, we suggest that instruction in problem solving should focus on having students identify the key features of given problems, and then practice manipulating the relationships between relevant variables.…”

Work in Progress: Testing an Assessment of Problem Solving in Introductory Chemical Process Design Courses

“…2. We also coded the responses to two questions for how closely student responses matched expert responses collected previously [19]. We first coded the criteria the students used to evaluate their process and how it matched the criteria listed by experts in the previous study.…”

“…Textbook problems are not suited to this task because the expert decisions are often made for the solver-for example, assumptions are almost always given to the solver in physics problems, rather than allowing the solver to identify appropriate assumptions or simplifications. We found that a troubleshooting task, such as critiquing a flawed product design schematic, was well-suited for this assessment, as it requires the solver to make many of the expert decisions [19]. The general structure for the assessment, which may be applied in any science and engineering discipline, is depicted in Fig.…”

“…We previously conducted a pilot study of this assessment to determine whether it reliably measures differences between more and less expert-like reasoning [19]. To evaluate students on problem-solving, we compared their assessment responses with a responses collected from experts.…”

Assessing authentic problem-solving in heat transfer

“…The basic components of the framework are 1) provide an authentic problem context, 2) ask a series of questions that require test-takers to make decisions about problem definition and planning how to solve, 3) provide more information, 4) ask a series of questions that require decisions about interpreting information and drawing conclusions, and 5) have test-takers choose and reflect on their solution. One of the authors has previously developed such an assessment in chemical process design 28 . One important feature of these assessments is that students are not graded based on a scale that compares them to one another, but rather compares their responses to a consensus of experts.…”