A regional R and D grant was used to explore how an informaticist could stimulate and respond to questioning behaviour in primary care clinicians. Training needs for the informaticist were identified and provided for. Nurses and doctors working in primary care were approached individually. They were encouraged to think about their clinical behaviour and to identify topics where they thought they might like to have more evidence either to justify their current practice or to stimulate change. The personal approach was labour intensive but seemed to provide a good yield in questions. The project took two years and the informaticists have tackled over 125 topics. The project is being evaluated by the Department of Primary Care at University College Hospital Medical School, London. A number of the GPs who used the service were interviewed and there was a high level of satisfaction. There was some informal feedback that the answers to the questions actually changed clinical behaviour but so far, audits of reported changes have not been carried out. Funding to continue and expand the service has now been secured through the access to evidence initiatives.

BackgroundOf 37 pediatric rheumatology fellowship training programs in the United States, many have three or fewer fellows at a given time, making large-scale assessment of fellow performance difficult. An objective structured clinical examination (OSCE) is a scenario-based simulation method that assesses individual performance, thus indirectly measuring training program effectiveness. This study describes the development and implementation of two national pediatric rheumatology OSCEs and methods used for programmatic improvement.MethodsOSCEs for pediatric rheumatology fellows were held in 2009 and 2011 during national rheumatology meetings using scenarios and assessment forms originally developed by a fellowship program director. The seven scenarios tested medical knowledge, physical exam and interpersonal skills. Pediatric rheumatologist evaluators assessed fellows’ performance using checklists and gave immediate feedback. Program directors were sent summaries of their fellows’ performances. Fellows evaluated the OSCE, providing organizational and scenario improvement suggestions. Programmatic changes to the 2011 OSCE were based on 2009 performance data and program evaluation feedback.ResultsTwenty-two fellows participated in 2009 and 19 in 2011. Performance scores in similar scenarios did not change considerably over the two iterations. In 2009, 85.7% of participants reported desire to change clinical behavior. Assessors’ 2009 program evaluation data prompted changes in rating scales and removal of invalid or unreliable assessments. Negative evaluation data about individual stations decreased from 60% in 2009 to 15.4% in 2011. Fellows’ ratings of the experience’s overall value were similar in 2009 and 2011. The average experience ratings were lower among fellows who proposed scenario-specific improvements and higher among those who recommended organizational improvements.ConclusionsThe 2011 examination exhibited programmatic improvement via reduction in fellows’ scenario-specific negative feedback. Fellows’ overall satisfaction did not change. Further work in scenario selection, assessment validation and inter-rater reliability will improve future pediatric rheumatology OSCEs.