Evaluation of Incremental Validity of Casper in Predicting Program and National Licensure Performance of Undergraduate Nursing Students: Protocol for a Mixed Methods Study (Preprint)

Abstract

BACKGROUND Academic success has been the primary criterion for admission to many nursing programs. However, academic success as an admission criterion may have limited predictive value for success in non-cognitive skills. Adding situational judgment tests, such as Casper, to admissions procedures may be one strategy to strengthen decisions and address the limited predictive value of academic admission criteria. The Bachelor Science in Nursing (Collaborative) Program at Memorial University is a competitive degree program. In 2021 the admissions processes were modified to include Casper based on concerns identified with non-cognitive skills. OBJECTIVE (1) To assess the incremental validity of Casper scores in predicting student performance at years 1, 2, 3, and 4 of the program and on National Council Licensing Exam (NCLEX) performance; (2) To examine faculty members’ perceptions of student performance and influences related to communication, professionalism, empathy, and problem-solving. METHODS We will use a multistage evaluation mixed methods study design with five phases with two embedded sequences. Phases 1-4 will align with performance in each year of the program, and the final phase will assess graduate performance on the national exam. Each phase will include an explanatory study sequence whereby quantitative data collection (i.e., student-completed questionnaires and psychomotor lab assessment results) will be followed by qualitative data collection (i.e., focus groups with faculty members). The goal of the focus groups is to help explain the quantitative findings and gain an enhanced understanding of the research problem. The study will also include an exploratory sequence whereby what is learned in each phase will inform focus group questions in the subsequent phase. All students enrolled in the first year of the nursing program were asked to participate (n=290). Faculty will be asked to participate in the focus groups at the end of each year of the program. Hierarchical multiple regression will be conducted for each outcome of interest to determine the extent that scores on Casper with admission grades, compared to admission grades alone, predict nursing student performance at years 1-4 of the program and success on the national exam. Thematic analysis of focus group transcripts will be conducted using Interpretive Description. The quantitative and qualitative data will be integrated after each phase is complete and at the end of the study. RESULTS The study was funded in September 2021, and data collection began in March 2022. Year 1 of data collection is complete, with 144 students enrolled and three faculty focus groups conducted. Year 1 data analysis and Year 2 data collection are in progress. CONCLUSIONS At the end of the study, we will provide the results of a comprehensive analysis to determine the extent that scores on Casper predict nursing student performance at years 1-4 of the program and on the NCLEX exam.