The development of systems thinking is considered a critical skill set for addressing interdisciplinary problems. This skill set is particularly important in the field of engineering, where engineers are often tasked with solving socio-technical problems that often require knowledge beyond their original discipline and practice in unfamiliar contexts. However, existing assessments often fail to accurately measure teachable knowledge or skills that constitute systems thinking. To investigate this issue, we compared students’ performance on two previously and independently peer-reviewed scenario-based assessments for systems thinking: The Village of Abeesee and the Lake Urmia Vignette. Twenty undergraduate engineering students participated in a multi-phase case study utilizing think aloud protocols and semi-structured interview methods to elicit the approaches students took thinking across the two instruments and past experiences that they felt prepared them to solve these ill-structured problems. We found that the way a scenario is presented to students impacts their subsequent problem-solving approach, which complicates assessment of systems thinking. Additionally, students identified only limited opportunities for the development of ill-structured problem-solving skills necessary for systems thinking. Our findings inform future work on improving systems thinking assessments and emphasize the importance of more intentionally supplying opportunities for students to practice solving ill-structured problems throughout the curriculum.