Content validity of a clinical problem solving test for use in recruitment to the acute specialties

Abstract

-Clinical problem solving tests (CPSTs) have been shown to be reliable and valid for recruitment to general practice (GP) training programmes. This article presents the results from a Department of Health-funded pilot into the use of a CPST designed for recruitment to the acute specialties (AS). The pilot paper consisted of 99 items from the validated GP question bank and 40 new items aimed specifically at topics of relevance to AS training. The CPST successfully differentiated between applicants. The overall test and the GP section showed high internal reliability, whereas the AS pilot section performed less well. A detailed item analysis revealed that the AS pilot items were, on average, more difficult and of poorer quality than the GP items. Important issues that need to be addressed in the early development phase of a test used for high stakes selection to specialty training programmes are discussed. Subsequent research has shown the machine marked tests (MMTs) used in GP selection are as valid and reliable as the current standard shortlisting method of scoring application form questions. 2 In light of this, this article presents the preliminary results from an ongoing Department of Health-funded pilot study of the development of a clinical problem solving test (CPST) tailored to the acute medical specialties, looking specifically at its reliability, content validity and face validity. The initial pilot testThe CPST paper consisted of 139 items mapped onto the foundation programme curriculum: 99 taken from the GP bank of validated items and 40 newly written items aimed specifically at topics of relevance to acute specialty (AS) training. The tried and tested GP items provided a useful benchmark against which to assess the performance of the newly developed AS items. The new items were written by a group of subject matter experts, including consultants in acute medicine, anaesthesia, emergency medicine and intensive care medicine.Applicants attending selection centres in 2008 at the South West Peninsula Deanery for CT1 training posts in anaesthesia, core medical training and the acute care common stem (ACCS) were invited to sit the first pilot CPST. Participation (or not) did not influence selection outcome in any way and participants consented to the linking of test results with other personal data. The paper was administered on the day of interview and an overall response rate of 74% (125 of 169 applicants) was achieved. Reliability of the testThe distribution of test scores was approximately normal in each section (GP and AS), indicating an absence of ceiling or floor effects and showing that the test has the potential to differentiate between candidates. Both the overall test and the GP item section showed very high internal reliability (Cronbach's ␣ϭ0.90 and 0.92 respectively). The AS pilot section performed less well (␣ϭ0.43), though this is partly attributable to the smaller number of items used. When corrected for test length (Spearman-Brown), the expected reliability of a 99-item test of equ...

Utility of selection methods for specialist medical training: A BEME (best evidence medical education) systematic review: BEME guide no. 45

“…Concerns with the construct validity of CPST have been raised. There was no firm evidence that the CPST validly tests problem-solving skills rather than knowledge (Patterson, Baron, et al 2009;Crossingham et al 2011). In pilot testing for selection processes into UK GP training, the CPST correlated with the SJT varying in the range of r ¼ 0.39 to r ¼ 0.53 (Patterson, Baron, et al 2009; (Patterson, Lievens, et al 2013).…”

Improving patient safety in the operating theatre and perioperative care: obstacles, interventions, and priorities for accelerating progress

“…patient outcomes) Reliability: inter-rater reliability, internal consistency Sensitivity: in relation to levels of performance (i.e. distinguishing poor from good performers) Transparency: people assessed understand the performance criteria against which they are being rated; availability of reliability and validity data Usability: simple framework, easy to train, easy to understand, easy to observe, domain-appropriate language, sensitive to rater workload Can provide a focus for training goals and needs Baselines for performance criteria are available and can be used appropriately by raters Minimal overlap between assessment components Evidence-based selection, using appropriately validated tasks and the concept of assessment/selection centres, is feasible across specialities, including acute care, 104 surgery, 105 106 and anaesthesia. 107 108 Gale and colleagues, 107 specifically, have shown correlations between performance within the assessment centre setting and job performance over the first year of the candidate's clinical appointment.…”

Opening the black box of selection

“…collaboration or empathy) are actually assessed in medical school selection procedures, and whether this is in line with what was intended from their outcome-based focus (Christian et al 2010;Wilkinson and Wilkinson 2016). This means that selection may be considered a sort of 'black box' (Kreiter 2017;Kulasegaram 2017;Lievens et al 2008), a paradoxical situation in which selection tools may predict outcomes but which constructs are actually doing the predicting is uncertain (Cleland et al 2014;Crossingham et al 2011;Tiller et al 2013). It is essential to know more about what is actually being measured (i.e.…”