Verbal Protocol Analysis as a Method to Document Engineering Student Design Processes

Abstract

Undergraduate engineering programs have faced numerous challenges in recent years. One of these challenges is to improve the way open‐ended design is taught. Although changes are underway in schools throughout the United States, not enough evaluation has been done to determine the impact of these changes. In this paper we describe a research tool that can also be used to assess student learning: verbal protocol analysis. In particular, this tool can be used to document the processes that engineering students use to solve open‐ended engineering design problems. The objective of this paper is to demonstrate the use of verbal protocol analysis as a method to assess student design processes. We also discuss the research questions that can be addressed by verbal protocol analysis and the opportunity to include this type of study as part of an engineering program evaluation.

Design Practices of Preservice Elementary Teachers in an Integrated Engineering and Literature Experience

“…Validated by the broad body of research on engineering design cognition and behavior (for a recent review, see Crismond & Adams, 2012), the Design Activity coding scheme includes codes for the phases of problem scoping (problem definition and information gathering), alternative solution development (idea generation, modeling, feasibility analysis, and evaluation), and project realization (decision and communication). Originally developed through a synthesis of textbooks commonly used in first-year engineering classes (Moore et al, 1995), the Design Activity coding scheme was applied to verbal protocol analysis of the design processes of individual college freshmen and seniors (Atman & Bursic, 1998;Atman, Chinka, Bursic, & Nachtmann, 1999). Later Atman and colleagues applied the framework to compare college seniors and practicing engineers completing the same individual design task (Atman, Adams, Mosborg, Cardella, Turns, & Saleem, 2007) and to study changes in college students' design processes from their freshman to senior years (Cardella, Atman, Turns, & Adams, 2008).…”

Analyzing Engineering Design through the Lens of Computation

“…Traditionally, work in engineering education and project-based learning has involved developing new approaches for assessing learning and knowledge. Typically, studies from this body of research focus on qualitative analyses of student language (Atman & Bursic, 1998;Dym, 1999;Russ, Scherr, Hammer, & Mikeska, 2008), student artifacts (Dong & Agogino, 1997;Lau, Oehlberg, & Agogino, 2009), or the combination of language and artifacts (Atman & Bursic, 1998,;Worsley & Blikstein, 2011) created in the process of designing, building and/or inventing. We look to contribute to the body of engineering education research by analyzing these practices at a very fine-grained scale.…”

“…More specifically, a collection of researchers has investigated design patterns on engineering tasks through think-alouds (Atman & Bursic, 1998;Ericsson & Simon, 1980;Russ et al, 2008). When considering expertise in the engineering context, many of the constructs discussed have been cast under different names: computational thinking (Resnick et al, 1998;Wing, 2006;Guzdial, 2008), designing thinking (Dym, 1999;Dym, Agogino, Eris, Frey, & Leifer, 2005), and mechanistic reasoning (Russ et al, 2008).…”

“…When considering expertise in the engineering context, many of the constructs discussed have been cast under different names: computational thinking (Resnick et al, 1998;Wing, 2006;Guzdial, 2008), designing thinking (Dym, 1999;Dym, Agogino, Eris, Frey, & Leifer, 2005), and mechanistic reasoning (Russ et al, 2008). Because each of these constructs could easily be the subject of an entire review, we will only mention them in passing as to indicate that these ideas have contributed to the analyses in this paper we will focus on a single body of literature by Atman and her collaborators (Adams, Turns and Atman, 2003;Atman & Bursic, 1998;Atman, Chimka, Bursic, and Nachtmann, 1999;Atman et al, 2007;Atman, Kilgore, & McKenna, 2008) and which is representative of the state of the field, and is directly related to our analyses. Atman and Bursic (1998), Atman et al (1999), and Adams, Turns, and Atman (2003) investigate engineering design language and practices by comparing engineering practices between freshmen and senior engineering students.…”

WORK IN PROGRESS: Quantitative Information Acquisition and Utilization by First-Year Engineering Students

“…For this study, we employed a verbal protocol methodology 7 whereby participants "think aloud" as they proceeded with their design. The entire design session is video and audio recorded for data analysis.…”