Comparing Self-Report Assessments and Scenario-Based Assessments of Systems Thinking Competence

Abstract

Self-report assessments are used frequently in higher education to assess a variety of constructs, including attitudes, opinions, knowledge, and competence. Systems thinking is an example of one competence often measured using self-report assessments where individuals answer several questions about their perceptions of their own skills, habits, or daily decisions. In this study, we define systems thinking as the ability to see the world as a complex interconnected system where different parts can influence each other, and the interrelationships determine system outcomes. An alternative, less-common, assessment approach is to measure skills directly by providing a scenario about an unstructured problem and evaluating respondents’ judgment or analysis of the scenario (scenario-based assessment). This study explored the relationships between engineering students’ performance on self-report assessments and scenario-based assessments of systems thinking, finding that there were no significant relationships between the two assessment techniques. These results suggest that there may be limitations to using self-report assessments as a method to assess systems thinking and other competencies in educational research and evaluation, which could be addressed by incorporating alternative formats for assessing competence. Future work should explore these findings further and support the development of alternative assessment approaches.

Exploring scenario‐based assessment of students' global engineering competency: Building evidence of validity of a China‐based situational judgment test

“…Although many of these critiques focus on self‐reported learning gains, many of the same concerns could potentially apply to self‐reported knowledge or competence. Our own work on engineering students' systems thinking competencies has shown that self‐reports align with one another but do not align with students' performance on multiple scenario‐based assessments of systems thinking (Davis et al, 2023). Along these lines, Anderson et al (2017) suggest that SJTs may help mitigate instances of stereotype threat and response‐style biases as compared to self‐report surveys as a result of their less test‐like structure and more interactive nature.…”

The Effects of a Humanities-Based Engineering Course on Engineering Students’ Sociotechnical Thinking

“…Student responses to the scenario-based assessment were scored on a scale of 0 to 3 for each dimension as per the rubric given in Mazzurco & Daniel (2020). For more details on the scenario, see Fig 1 . We chose to use a scenario-based assessment because these assessments allow some insight into students' thinking and may more directly measure students' abilities than traditional selfreport assessments (Davis et al, 2023). Furthermore, scenariobased assessments can be used as an instructional tool as well as an assessment tool (Davis et al, 2023).…”

“…For more details on the scenario, see Fig 1 . We chose to use a scenario-based assessment because these assessments allow some insight into students' thinking and may more directly measure students' abilities than traditional selfreport assessments (Davis et al, 2023). Furthermore, scenariobased assessments can be used as an instructional tool as well as an assessment tool (Davis et al, 2023). The course we studied for this project used case studies to teach students to Functionality: Efficiency, feasibility, ease of operation, maximum power generated, friction, storage of energy, functioning of components, alternative techs to meet the same goals, ability to generate the needed energy output, and so forth.…”

Identification of Complex Thinking Related Competencies

“…A statistical analysis of the factors influencing the acquisition of complex thinking would give educators tools to better plan their curricula, courses, and lessons. Previous studies have approached student development of complex thinking with self-reporting assessments (Davis et al, 2023), pre-test and post-test comparisons (Gago et al, 2020), and interview questions (Lizier et al, 2018). In our study, we propose using statistical methods to better understand the relationship among the subcompetencies, which could influence curriculum design in the future.…”