Assessing learning processes rather than outcomes: using critical incidents to explore student learning abroad

Abstract

There is an increasing emphasis on assessing student learning outcomes from study abroad experiences, but this assessment often focuses on a limited range of outcomes and assessment methods. We argue for shifting to assessing student learning processes in study abroad and present the critical incident technique as one approach to achieve this goal. We demonstrate this approach in interviews with 79 students across a range of global engineering programs, through which we identified 173 incidents which were analyzed to identify common themes. This analysis revealed that students described a wide range of experiences and outcomes from their time abroad. Students’ experiences were messy and complex, making them challenging to understand through typical assessment approaches. Our findings emphasize the importance of using a range of assessment approaches and suggest that exploring students’ learning processes in addition to learning outcomes could provide new insights to inform the design of study abroad programs.

Improving Science Learning Outcomes Through the Implementation of the Inquiry Learning Model During the Covid-19 Pandemic in Class IX-A Students of SMP Negeri 1 Bendungan for the 2021/2022 Academic Year

“…Things that need to be improved are by rearranging the Learning Implementation Plan in class, placing emphasis again on learning activities, including encouraging students. to ask questions actively and follow all stages of learning well, especially for students who have not had improved learning outcomes in the pre-cycle or first cycle, providing encouragement to look for references and explore the literature in order to obtain complete knowledge when collecting data to test hypotheses (Davis & Knight, 2023). The technique of dividing roles is also maximized in groups, both when collecting data and during group presentations.…”

Bridging borders: assessing the impact of semester-long study abroad programs on intercultural competence development in undergraduate engineering students

“…The results of the current study are consistent with other studies that focused on assessing the effectiveness of intentionally structured study abroad initiatives. For example, the study by Bittinger et al (2022) and Davis and Knight (2022) has emphasized how intentionally structured study abroad programs can bring about changes in the intercultural competence of students. The results of Krishnan et al (2017) and Bittinger et al (2022) are consistent with the overall results of our study, where we also witnessed that, as whole, engineering students demonstrated a significant and meaningful gain in intercultural learning.…”

“…From the demographic data, we note that for the C3 the percentage of domestic students, and male students was larger as compared to other clusters, therefore, demographic and personal factors highlighted in prior literature could be the reason for the decline in intercultural competence shown by the IDI for C3 students. Streitwieser and Light (2018) have argued that study abroad experiences are messy, therefore capturing the student experiences and intercultural development only through a IDI survey is not enough, therefore we need to take a deeper plunge into the experiences by introducing some qualitative techniques (Davis & Knight, 2022). Moreover, to triangulate our findings and to ground our choice of methods as proposed in recent literature (see, Tamer et al, 2023), we conducted qualitative analysis of the final reflections along with assessing the intercultural learning gains using the IDI.…”

Exploring scenario‐based assessment of students' global engineering competency: Building evidence of validity of a China‐based situational judgment test

“…Although this study suggests that scores on the self‐report instruments correlate with students' time abroad, our findings do not indicate that they are measuring competence or knowledge, at least as measured by a scenario‐based assessment. We do not have evidence yet to suggest that the GEC China SJT measures these outcomes either, but using multiple methods can provide program evaluators with a more holistic understanding of what students are experiencing and learning through their programs (Davis & Knight, 2023).…”