Students' experimentation strategies in design: Is process data enough?

Abstract

Conducting experiments is an important practice for both engineering design and scientific inquiry. Engineers iteratively conduct experiments to evaluate ideas, make informed decisions, and optimize their designs. However, both engineering design and scientific experimentation are open‐ended tasks that are not easy to assess. Recent studies have demonstrated how technology‐based assessments can help to capture and characterize these open‐ended tasks using unobtrusive data logging. This study builds upon a model to characterize students' experimentation strategies in design (ESD). Ten undergraduate students worked on a design challenge using a computer‐aided design (CAD) tool that captured all their interactions with the software. This “process data” was compared to “think‐aloud data,” which included students' explanations of their rationale for the experiments. The results suggest that the process data and the think‐aloud data have both affordances and limitations toward the goal of assessing students' ESD. While the process data was an effective approach to identify relevant sequences of actions, this type of data failed to ensure that students carried them out with a specific purpose. On the other hand, the think‐aloud data captured students' rationale for conducting experiments, but it depended on students' ability to verbalize their actions. In addition, the implementation of think‐aloud procedures and their analysis are time consuming tasks, and can only be done individually.

A systematic review of empirical studies using log data from open‐ended learning environments to measure science and engineering practices

“…Some of the OELEs are associated with standardized assessments, such as the Programme for International Student Assessment (PISA) and the National Assessment of Educational Progress (NAEP) (eg, PISA (Teig et al, 2020), NAEP (Chu & Leighton, 2019)). Furthermore, some OELEs are game‐like, wrapping the scenarios to explore in a story (eg, Crystal Island (Taub et al, 2018), VPA (Jiang et al, 2015), Energy3D (Vieira et al, 2018), TugLet (Käser et al, 2017), BioWorld (Doleck et al, 2016a)), while other OELEs are more closely associated with virtual labs (eg, MicroDYN (Greiff et al, 2016), PhET (Wang, Salehi, et al, 2021), Inq‐ITS (Gobert et al, 2012) and NetLogo (Southavilay et al, 2013)). Lastly, there is also a teachable agent game, Betty's Brain (Segedy et al, 2015b).…”

“…At the same time, aggregated features based on raw actions alone were found to have limited predictive power for target competencies (Fratamico et al, 2017; Pedro et al, 2012; Wang, Salehi, et al, 2021). One way to address this challenge is through combining sub‐sequences of discrete actions into semantically meaningful units, or events, and counting the instances of these units, such as experiment trials set up by students to collect data (Vieira et al, 2018) or test hypotheses (Gobert et al, 2013). These features are informed by researchers' knowledge of theoretical frameworks or observations of students working through the task, thus better aligned with the underlying cognitive processes.…”

Data science knowledge integration: Affordances of a computational cognitive apprenticeship on student conceptual understanding

“…Consequently, DS has become a crucial tool for the research and educational community [3]. Educational research has explored diverse topics using DS tools, including student admission [72] and retention [71], as well as student approaches to problem‐solving [65]. However, less is known about the processes involved in DS education itself, with the persistent question still being “How do we train [a] workforce of professionals who can use data to its best advantage?” (p.2) [3].…”

Reflection in Time: Using Data Visualization to Identify Student Reflection Modes in Design

“…Given the complexity of the design process, visualization techniques are commonly deployed to both characterize designers' processes and enable comparisons of how designers performed on a given task [21][22][23][24][25][26] . These visualizations may seek to visualize a designer's entire design process or relevant aspects of their design process.…”