Development and Validation of the Engineering Computational Thinking Diagnostic for Undergraduate Students

Abstract

Computational thinking is one barrier to enculturating as a professional engineer. We created the Engineering Computational Thinking Diagnostic (ECTD) as an instructional tool that can identify at-risk first-year engineering students. The purpose of this study is to provide construct validity, internal consistency reliability, item characteristics, and criterion validity evidence for this diagnostic. From fall 2020 to fall 2021, 469 students from three institutions in the United States took the diagnostic. The data from 152 students at one institution was used to provide evidence of predictive validity. Exploratory and confirmatory factor analyses resulted in 20 items loading onto one factor in a good model fit range, with the internal consistency reliability coefficient, Cronbach α of 0.86. From item analyses based on classical test theory, the diagnostic items on average tended to be slightly easy but had sufficient discrimination power. The correlation matrix for criterion validity evidence indicated that the diagnostic functions well to differentiate students' computational thinking ability by prior computer science course experience as well as by first-generation status. Predictive validity evidence from regression analyses revealed the statistically significant effect of students' diagnostic scores assessed at the beginning of the first semester on predicting their end of semester course grades. The ECTD can have a broad impact because it provides a tool to gauge the entry-level skills of students, enabling early curriculum interventions to help retention and persistence to graduation. We make the case that the ECTD could contribute to the development of a more diverse workforce in engineering.

Board 240: Computational Thinking in the Formation of Engineers: Year 3

“…Approximately 900 students across the three institutions completed ECTD testing at the start of the 2021-2022 academic year. Our consulting psychometrician used this data to complete confirmatory factor analysis on the ECTD [23]. This confirmatory factor analysis fully validated the ECTD as predictive of student performance in classes containing computational thinking and computer programming.…”