Evaluation of Situational Judgment Tests in student selection in Indonesia and the impact on diversity issues

Abstract

Background Internationally, medical selection relies heavily on prior academic attainment which has an adverse impact on the diversity of selected students. Since non-academic attributes are also important, this study aims to evaluate the use of a Situational Judgment Test (SJT) for selection and the impact on student diversity relating to gender, ethnicity and socio-economic status. Previous SJT research has almost entirely originated from a Western context and this study focuses on new evidence in a South East Asian context with a different demographic profile. Methods Thirty faculty members developed 112 SJT scenarios assessing professionalism, communication and self-awareness domains. The scenarios underwent a concordance stage where stakeholder input was sought on the content appropriateness, to define the item scoring key, followed by an initial psychometric evaluation with first and second year medical students (N = 436). Based on these results, 30 scenarios, consisting of 128 nested items, were selected for pilot testing and evaluation regarding diversity issues with two cohorts of applicants in 2017 (N = 446) and 2018 (N = 508). Results The SJT demonstrated good internal consistency (Cronbach’s alpha of 0.80 and 0.81 respectively). There were significant differences in SJT scores based on gender in both years, where females consistently outperformed males (p = .0001). However, no significant differences were found based on high school origin, parental educational background or ethnicity. Conclusions This is the first study to evaluate the use of an SJT in Indonesia, which has a unique diversity profile compared to Western countries. Largely, the preliminary results replicate previous studies of the potential diversity benefits of using an SJT as a tool for medical student selection and has the potential to level the playing field regarding socio-economic status and ethnicity. Further studies exploring more variables representing diversity are warranted to confirm the early results in this study.

Performance Prediction of the Family Medicine Applicants of Oman Medical Specialty Board in Multiple Mini-interviews Using the International Foundations of Medicine Exam and Situational Judgment Test

“…In another study, Patterson et al confirmed the reliability of SJTs in medical selection, predicting performance in different medical education settings (undergraduate and postgraduate) and their incremental validity over other selection methods (9). Furthermore, the variations in SJT validity, reliability, and diversity benefits are attributed to the cultural differences between Western and Asian contexts concerning professionalism, values, and communication style (27).…”

Urgensi Pemanfaatan Metode Seleksi Atribut Non-Kognitif Dalam Sistem Seleksi Calon Mahasiswa Kedokteran Tahap Undergraduate

“…Atau tidak memberi tugas kepada teman tersebut?). 11 Melalui SJT dapat diketahui apa yang menurut calon mahasiswa sebagai respons paling sesuai atau apa yang akan calon lakukan jika nantinya dihadapkan pada situasi serupa di dunia nyata. Apakah calon sungguh-sungguh akan melakukan opsi yang dia pilih tersebut tidak dapat diketahui.…”

“…Penggunaan SJT di Indonesia masih terbatas namun penelitian Soemantri et al 11 memperlihatkan potensi SJT sebagai salah satu metode seleksi calon mahasiswa berdasarkan atribut non-kognitif. Uji coba SJT yang dilakukan pada penelitian tersebut menilai atribut profesionalisme, komunikasi efektif dan sikap mawas diri calon mahasiswa kedokteran.…”