Psychometric characteristics of simulation‐based assessment in anaesthesia and accuracy of self‐assessed scores*

Weller, Jennifer; Robinson, Brian; Jolly, Brian; Watterson, LM; Joseph, Michele; Bajenov, S; Haughton, AJ; Pd, Larsen

doi:10.1111/j.1365-2044.2004.04073.x

Cited by 54 publications

(45 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The fact that many participants successfully avoided most of the errors suggests that few individual aspects of the simulations were unreasonably challenging. The engagement of participants was palpably genuine and they rated the simulations highly for realism, but we acknowledge limitations to this realism [31]. We think fidelity is critical to research involving immersion simulation, and that there is room for improvement in currently available simulators, but we believe we have paid as much attention to detail as we could (for example by using real drugs).…”

Section: Discussionmentioning

confidence: 97%

See 1 more Smart Citation

A simulation design for research evaluating safety innovations in anaesthesia*

et al. 2008

Self Cite

View full text Add to dashboard Cite

SummaryIt is notoriously difficult to obtain evidence from clinical randomised controlled trials for safety innovations in healthcare. We have developed a research design using simulation for the evaluation of safety initiatives in anaesthesia. We used a standard and a modified scenario in a human-patient simulator, involving a potentially life-threatening problem requiring prompt attention -either a cardiac arrest or a failure in oxygen supply. The modified scenarios involved distractions such as loud music, a demanding and uncooperative surgeon, telephone calls and frequent questions from a medical student. Twenty anaesthetics were administered by 10 anaesthetists. A mean (SD) of 11.3 (2.8) errors per anaesthetic were identified in the oxygen failure scenarios, compared with 8.0 (3.4) in the cardiac arrest scenarios (ANOVA: p = 0.04). The difference between the combined standard scenarios and the combined modified scenarios was not significant. The mean rate of errors overall was 9.7 per simulation, with a pooled SD of 4.46, so in future studies 21 subjects would provide 80% statistical power to show a reduction in error rate of 30% from baseline with p £ 0.05. Our research design will facilitate the evaluation of safety initiatives in anaesthesia.

show abstract

Section: Discussionmentioning

confidence: 97%

“…Simulation can potentially be used to assess anaesthetic practice or equipment [26][27][28][29][30][31][32]. The CPS has a number of advantages over the clinical setting for the study of human error.…”

mentioning

confidence: 99%

A simulation design for research evaluating safety innovations in anaesthesia*

et al. 2008

Self Cite

View full text Add to dashboard Cite

show abstract

“…Residents in the upper quartile of performance underestimated their performance in additional specific competencies, whereas residents in the lowest performance quartile overestimated professionalism skills. A similar study in anesthesia residents demonstrated moderate correlation between self-and observer assessments when reviewing their performance on three emergency HFS scenarios; however, this correlation was poorer at the lower levels of performance, 97 further supporting the unreliability of self-assessment for patient care competence.…”

Section: Overview Of Assessment Methods Identifiedmentioning

confidence: 96%

Assessing Patient Care: Summary of the Breakout Group on Assessment of Observable Learner Performance

Takayesu

Kulstad

Wallenstein

et al. 2012

Academic Emergency Medicine

View full text Add to dashboard Cite

There is an established expectation that physicians in training demonstrate competence in all aspects of clinical care prior to entering professional practice. Multiple methods have been used to assess competence in patient care, including direct observation, simulation-based assessments, objective structured clinical examinations (OSCEs), global faculty evaluations, 360-degree evaluations, portfolios, self-reflection, clinical performance metrics, and procedure logs. A thorough assessment of competence in patient care requires a mixture of methods, taking into account each method's costs, benefits, and current level of evidence. At the 2012 Academic Emergency Medicine (AEM) consensus conference on educational research, one breakout group reviewed and discussed the evidence supporting various methods of assessing patient care and defined a research agenda for the continued development of specific assessment methods based on current best practices. In this article, the authors review each method's supporting reliability and validity evidence and make specific recommendations for future educational research.ACADEMIC EMERGENCY MEDICINE 2012; 19:1379-1389 by the Society for Academic Emergency Medicine I n 2001, the Accreditation Council for Graduate Medical Education (ACGME) introduced a timeline for the implementation of training and assessment in six core competencies that form the foundation of clinical competence. Introduced in 1996, the Canadian CanMEDS manager competency correlates to the AC-GME patient care competency, broadly defined as "the active engagement in decision-making in the operation of the healthcare system."1 The patient care competency for emergency medicine (EM) has been defined by a previous Academic Emergency Medicine (AEM) consensus conference, 2 now further elaborated on by the milestones in training, 3 as being able to efficiently gather and synthesize medical and diagnostic information, prioritize tasks, and implement management plans on multiple patients, as well as performing essential invasive procedures competently.There is an explicit expectation that physicians in training demonstrate competence in various aspects of clinical care prior to graduation and professional practice. 4 While this accountability falls squarely on the shoulders of residency training programs, it is mirrored by commensurate expectations of maintenance of competency during ongoing professional practice.The goals of the 2012 AEM consensus conference patient care working group were to describe the current state of evidence for assessment of competence in patient care and define a research agenda for the further development of specific assessment methods based on current best practices. METHODSA search was conducted using MEDLINE 1996-present using the key word search terms "assessment," "patient care," "competency," "competence," "assess*," "emergency," and "education" and limited to humans and English language [boolean search: ((assessment and patient care AND (competency or competence)) OR (assess* a...

show abstract

“…Weller and colleagues revealed that anesthesiology residents' self-ratings of simulation-based performance modestly correlated with those of external assessors and that lower-scoring residents overrated their performance compared to higher-scoring residents (5). MacDonald showed, in a cohort of medical students performing technical skills, that with repetitive self-assessment, accuracy of self-assessment increased (6).…”

Section: Introductionmentioning

confidence: 96%