Validity: applying current concepts and standards to gynecologic surgery performance assessments

Abstract

Validity is critical for meaningful assessment of surgical competency. According to the Standards for Educational and Psychological Testing, validation involves the integration of data from well-defined classifications of evidence. In the authoritative framework, data from all classifications support construct validity claims. The two aims of this study were to develop a categorization method for validity evidence published in support of surgery performance assessments and to summarize the results of applying this methodology to the gynecologic surgery literature. This was a critical analysis of published observations reported as validity evidence in studies with a construct validity claim. Medline and Embase databases were searched using keywords: "surgery" and "construct validity". Parameters included English-language articles published from 2000 to 2012. Gynecologic studies were analyzed for definitions of construct validity and nonstandard terminology. Categorization criteria were developed and applied by the researchers to all observations. Two independent evaluators examined reported observations for compliance with guidelines provided by the Standards. Inter-rater agreement was calculated using weighted kappa. The initial search returned 167 articles. Twenty-five articles were left for inclusion in our analysis. Eighteen (72 %) articles defined construct validity as the ability to discriminate between expert and novice levels of proficiency. Within the sample, 80 discrete observations of reported validity evidence were identified and categorized according to standard classifications. Nearly 30 % of all published observations intended to demonstrate differences in performance by level of proficiency, 25 % described a scoring model, and 14 % demonstrated support of assessment content. Not one article contained a statistical correlation between assessment scores and objective outcomes from the authentic surgical environment. Medians for level of rigor ranged from 0 to 1 across all forms of evidence. Weighted kappa values ranged 0.60-0.91. Validity claims in gynecologic surgical assessment over-rely on generalizability evidence. No test-criterion evidence was observed. Increased awareness of current standards and systematic argument development is needed for gynecologic performance assessments.

Decisions and Consequences: Validation of High-Stakes Simulation-Based Assessments in Gynecologic Surgery

“…It is important to note the limited educational value of an assessment that, statistically, can only distinguish between early novices (junior trainees) and advanced experts (senior faculty). To date, there has been an overreliance on expert-novice comparison data as the basis of validation claims in surgical performance assessment [14]. Other sources of robust generalizability evidence include comparisons of blinded assessments on a simulation-based performance with in-vivo performance (e.g., video from the operating room).…”

Diagnostic Radiology Residency Assessment Tools: A Scoping Review

“…Global evaluations of assessment tools in multiple specialities have been reported in internal medicine, obstetrics, and anesthesia, all seeking to evaluate areas of deficiency in the current assessment tools, and providing evidence for their use. 26 -29…”

“…We modeled our scale after similar articles in the medical literature evaluating assessment tools in Obstetrics and Internal medicine. 26,27 The article was initially examined for any explicit definition of validity. Subsequently, the frequency and quality of evidence was judged for each assessment tool with a rating scale for each of the 5 domains: “0—no evidence,” “1—low rigor: evidence establishing weak compliance with standards,” “2—moderate to high rigor: evidence establishing strong compliance with standards.” Detailed rating justifications were documented in a second open text table.…”

Developing expertise in gynecologic surgery: reflective perspectives of international experts on learning environments and processes

“…Based on its applied nature, surgery has been referred to as “a craft”,33,34 but expertise in highly technical professions is more complex than such a term communicates. Depending on the precise nature of the work, and its process and products, expertise is defined in a number of ways 9,10.…”

“…Historically, much of the existing research on medical expertise has been done primarily with a few “experts” (often defined as experts primarily by time-since-graduation) 34,41. They are often compared to convenience samples of medical students without authentic experience in practice 42.…”