Developing a situational judgment test blueprint for assessing the non-cognitive skills of applicants to the University of Utah School of Medicine, the United States

Abstract

Purpose:The situational judgment test (SJT) shows promise for assessing the non-cognitive skills of medical school applicants, but has only been used in Europe. Since the admissions processes and education levels of applicants to medical school are different in the United States and in Europe, it is necessary to obtain validity evidence of the SJT based on a sample of United States applicants.Methods:Ninety SJT items were developed and Kane’s validity framework was used to create a test blueprint. A total of 489 applicants selected for assessment/interview day at the University of Utah School of Medicine during the 2014-2015 admissions cycle completed one of five SJTs, which assessed professionalism, coping with pressure, communication, patient focus, and teamwork. Item difficulty, each item’s discrimination index, internal consistency, and the categorization of items by two experts were used to create the test blueprint.Results:The majority of item scores were within an acceptable range of difficulty, as measured by the difficulty index (0.50-0.85) and had fair to good discrimination. However, internal consistency was low for each domain, and 63% of items appeared to assess multiple domains. The concordance of categorization between the two educational experts ranged from 24% to 76% across the five domains.Conclusion:The results of this study will help medical school admissions departments determine how to begin constructing a SJT. Further testing with a more representative sample is needed to determine if the SJT is a useful assessment tool for measuring the non-cognitive skills of medical school applicants.

Computerized test versus personal interview as admission methods for graduate nursing studies: A retrospective cohort study

“…This meritocratic approach is widely accepted in academic institutions in Israel and in other countries (Linn, ; Creech & Aplin‐Kalisz, ). However, these types of selection methods fail to address other requirements of the future professional, such as non‐cognitive skills, including effective communication, teamwork, ethical behavior, and displaying the highest level of professionalism (Dahlin et al ., ; Colbert‐Getz et al ., ).…”

Exploring Behavioral Competencies for Effective Medical Practice in Nigeria

“…The scoping review resulted in a starting list of 21 behavioral competencies defined in Table 1: collaboration, effective communication, resilience, professionalism, management, empathy, health advocacy, patient-centeredness, ethical responsibility, role differentiation, social affect, cultural sensitivity, systems-based practice, neutrality, probity, flexibility, initiative, creativity, reliability, service orientation, and capacity for improvement. 10,11,[22][23][24][25][26][27][28] Following round robin contributions, the NGT with medical students and doctors yielded 63 and 42 competencies respectively. Elimination of overlapping concepts condensed the list of behavioral competencies to 32 for medical students and 27 for medical doctors.…”

“…The starting list for the discussion was derived from the results of the scoping review on the global evidence on behavioral competencies required for medical practice. 10 , 11 , 22 - 28 These competencies were grouped by the participants based on their relevance for interactions with clients or colleagues. During the NGT exercise with resident doctors, this starting list had been modified to reflect the list of competencies generated from the first NGT exercise with the medical students.…”

Non-technical attributes in paramedicine: Is situational judgement testing the solution?

“…(3,15) Lievens et al (2006) argues this could be due to the ability to collect information on facial expressions, body language and voice inclinations from video based SJT's, which play a significant role in interpersonal interactions. (15) Further research has been conducted using alternate delivery methods including computer-based SJTs (16) and iPads (17). Both of these were found to be effective and would further alleviate the resource intensiveness required to score SJTs.…”