Validity Evidence and Scoring Guidelines for Standardized Patient Encounters and Patient Notes From a Multisite Study of Clinical Performance Examinations in Seven Medical Schools

Park, Yoon Soo; Hyderi, Abbas; Heine, Nancy; May, Win; Nevins, Andrew; Liu, Ming; Bordage, Georges; Yudkowsky, Rachel

doi:10.1097/acm.0000000000001918

Cited by 18 publications

(19 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As noted above, we observed variability in methodological choices. In many instances, authors used psychometric principles, measurement concepts, and statistical conventions in their work (e.g., Cameron et al, 2017;DeMuth et al, 2018;Dore et al, 2017;Hauer et al, 2018;Li et al, 2017Li et al, , 2018Naidoo et al, 2017;Park et al, 2016Park et al, , 2017. By contrast, we also noted methods associated with qualitative paradigms (e.g., Pool et al, 2018), though comparatively fewer.…”

Section: Level 2-summarizing Our Interpretations and Findings When Attempting To Appraise Compatibilitymentioning

confidence: 87%

“…Fewer authors appeared to leverage constructivist/interpretivist framings (e.g., Christensen et al, 2018;Pool et al, 2018). We noted instances in which observers were considered objective, but fallible, interchangeable and as contributing error that could be mitigated through training (e.g., Biagioli et al, 2017;Cameron et al, 2017;Dory et al, 2018;Naumann et al, 2016;Park et al, , 2016Park et al, , , 2017Roberts et al, 2017a, b;Turner et al, 2017;Weingart et al, 2018). By contrast, others appeared to value observer subjectivity, and positioned the variation between observers as meaningful (e.g., Chahine et al, 2016;Christensen et al, 2018;Pool et al, 2018).…”

Section: Differences In the Way Assessment Features Are Enacted: Suggesting Variable Positionalitymentioning

confidence: 98%

“…This consequently led us to focus on how authors reported enacting assessment features, from which we presumed their positionality. For example, we interpreted some authors to associate their operationalization of constructs with post-positivist positions (e.g., as latent traits or attributes) (e.g., Bajwa et al, 2017;Biagioli et al, 2017;Cameron et al, 2017;DeMuth et al, 2018;Hicks et al, 2016;Lee et al, 2016;Li et al, 2017Li et al, , 2018Onishi et al, 2018;Park et al, 2016Park et al, , 2017Ross et al, 2017). Fewer authors appeared to leverage constructivist/interpretivist framings (e.g., Christensen et al, 2018;Pool et al, 2018).…”

Section: Differences In the Way Assessment Features Are Enacted: Suggesting Variable Positionalitymentioning

confidence: 99%

“…By contrast, others appeared to value observer subjectivity, and positioned the variation between observers as meaningful (e.g., Chahine et al, 2016;Christensen et al, 2018;Pool et al, 2018). A common approach to validation included leveraging statistical approaches (e.g., generalizability, reliability, controlling for rater effects) (e.g., Cameron et al, 2017;Chahine et al, 2016;DeMuth et al, 2018;Dory et al, 2018;Li et al, 2018;Mink et al, 2018;Onishi et al, 2018;Park et al, , 2016Park et al, , , 2017Roberts et al, 2017a, b). Fewer authors used qualitative forms of justification (e.g., credibility, member-checking, transferability) (e.g., Christensen et al, 2018;Pool et al, 2018).…”

Section: Differences In the Way Assessment Features Are Enacted: Suggesting Variable Positionalitymentioning

confidence: 99%

See 3 more Smart Citations

Implicit and inferred: on the philosophical positions informing assessment science

Tavares

Hodwitz

Rowland

et al. 2021

Adv in Health Sci Educ

View full text Add to dashboard Cite

Assessment practices have been increasingly informed by a range of philosophical positions. While generally beneficial, the addition of options can lead to misalignment in the philosophical assumptions associated with different features of assessment (e.g., the nature of constructs and competence, ways of assessing, validation approaches). Such incompatibility can threaten the quality and defensibility of researchers' claims, especially when left implicit. We investigated how authors state and use their philosophical positions when designing and reporting on performance-based assessments (PBA) of intrinsic roles, as well as the (in)compatibility of assumptions across assessment features. Using a representative sample of studies examining PBA of intrinsic roles, we used qualitative content analysis to extract data on how authors enacted their philosophical positions across three key assessment features: (1) construct conceptualizations, (2) assessment activities, and (3) validation methods. We also examined patterns in philosophical positioning across features and studies. In reviewing 32 papers from established peer-reviewed journals, we found (a) authors rarely reported their philosophical positions, meaning underlying assumptions could only be inferred; (b) authors approached features of assessment in variable ways that could be informed by or associated with different philosophical assumptions; (c) we experienced uncertainty in determining (in)compatibility of philosophical assumptions across features. Authors' philosophical positions were often vague or absent in the selected contemporary assessment literature. Leaving such details implicit may lead to misinterpretation by knowledge users wishing to implement, build on, or evaluate the work. As such, assessing claims, quality and defensibility, may increasingly depend more on who is interpreting, rather than what is being interpreted.

show abstract

Section: Level 2-summarizing Our Interpretations and Findings When Attempting To Appraise Compatibilitymentioning

confidence: 87%

Section: Differences In the Way Assessment Features Are Enacted: Suggesting Variable Positionalitymentioning

confidence: 98%

Section: Differences In the Way Assessment Features Are Enacted: Suggesting Variable Positionalitymentioning

confidence: 99%

Section: Differences In the Way Assessment Features Are Enacted: Suggesting Variable Positionalitymentioning

confidence: 99%

See 2 more Smart Citations

Implicit and inferred: on the philosophical positions informing assessment science

Tavares

Hodwitz

Rowland

et al. 2021

Adv in Health Sci Educ

View full text Add to dashboard Cite

show abstract

“…The articulation of argument-based validity by Kane [20], [21], although it actually referred to Cronbach [22], brought about a new view of validity and triggered many important studies in various disciplines or fields such as English as a foreign language [23], [24], Medicine [25], [26], and Mathematics [9], among others. Importantly, several types of validity came to an end [27] given the unified view.…”

Section: Traditional and Contemporary Views Of Validitymentioning

confidence: 99%

Some Notes on the Contemporary Views of Validity in Psychological and Educational Assessment

Widodo¹

2018

Proceedings of the 5th International Conference on Community Development (AMCA 2018)

View full text Add to dashboard Cite

This paper aims at providing an analysis of views of validity as an important concept in psychological and educational assessment which has undergone transformation for several decades. The revolutionary change took place along with the release of the 1999 Standards followed by the 2014 Standards in which validity is associated with interpretation and use of test scores. With the new framework, several types of theoretical and empirical evidence are required to support the validation process. The results of the analysis show that the drastic change of the validity concept and the complicated process of validation have resulted in misunderstanding, misinterpretation, and misuse. Apart from different views which challenge the new concept, the argument-based approach has provided basis for new studies in many fields. Some strategies required to deal with complicated validation processes are suggested, along with some alternative perspective associated with the limitations of the standards.

show abstract

Comparing Students’ Clinical Grades to Scores on a Standardized Patient Note-Writing Task

et al. 2020

View full text Add to dashboard Cite

BACKGROUND: Few assessments capture the diagnostic impressions medical students form immediately following patient encounters. However, notes written for objective structured clinical examinations (OSCEs) allow learners to document their clinical reasoning in real time. The University of Illinois at Chicago College of Medicine (UIC-COM) has developed a rubric for scoring patient notes (PNs) in their OSCE for senior students. OBJECTIVE: To validate the UIC-COM PN Scoring Rubric as a measure of clinical reasoning by comparing PN scores from a similar exam at the Columbia University Vagelos College of Physicians and Surgeons (VP&S) to clinical rotation performance. DESIGN: Cross-sectional analysis. PARTICIPANTS: From a total of 146 third-year medical students who completed the OSCE at VP&S in spring 2017, we selected 60 at random, 20 from each tertile of clinical rotation performance. MAIN MEASURES: We scored these students' PNs using the rubric's four sections-Documentation, Differential Diagnosis, Justification, and Workup, each scored from 1 to 4-and calculated a composite score (maximum 100). We used one-way ANOVA to examine differences in scores between clinical rotation performance tertiles. KEY RESULTS: Students in the bottom, middle, and top clinical rotation performance tertiles had mean Documentation scores of 2.54, 2.63, and 2.88, respectively (p = 0.02, bottom vs. top tertile). Mean composite scores were 61.98, 64.05, and 67.86, respectively (p = 0.02, bottom vs. top tertile). CONCLUSIONS: We showed an association between PN scores and clinical rotation performance. Since clinical rotation grades incorporate multiple types of assessments of students' clinical reasoning skills, we believe that this correlation lends validity evidence to using the notewriting task as a measure of clinical reasoning. Future directions include expanding the task to different stages of learners, to real life patient encounters, and to formative rather than summative assessments of note-writing skills.

show abstract

Validity Evidence and Scoring Guidelines for Standardized Patient Encounters and Patient Notes From a Multisite Study of Clinical Performance Examinations in Seven Medical Schools

Cited by 18 publications

References 20 publications

Implicit and inferred: on the philosophical positions informing assessment science

Implicit and inferred: on the philosophical positions informing assessment science

Some Notes on the Contemporary Views of Validity in Psychological and Educational Assessment

Comparing Students’ Clinical Grades to Scores on a Standardized Patient Note-Writing Task

Contact Info

Product

Resources

About