From aggregation to interpretation: how assessors judge complex data in a competency-based portfolio

Abstract

While portfolios are increasingly used to assess competence, the validity of such portfolio-based assessments has hitherto remained unconfirmed. The purpose of the present research is therefore to further our understanding of how assessors form judgments when interpreting the complex data included in a competency-based portfolio. Eighteen assessors appraised one of three competency-based mock portfolios while thinking aloud, before taking part in semi-structured interviews. A thematic analysis of the think-aloud protocols and interviews revealed that assessors reached judgments through a 3-phase cyclical cognitive process of acquiring, organizing, and integrating evidence. Upon conclusion of the first cycle, assessors reviewed the remaining portfolio evidence to look for confirming or disconfirming evidence. Assessors were inclined to stick to their initial judgments even when confronted with seemingly disconfirming evidence. Although assessors reached similar final (pass–fail) judgments of students’ professional competence, they differed in their information-processing approaches and the reasoning behind their judgments. Differences sprung from assessors’ divergent assessment beliefs, performance theories, and inferences about the student. Assessment beliefs refer to assessors’ opinions about what kind of evidence gives the most valuable and trustworthy information about the student’s competence, whereas assessors’ performance theories concern their conceptualizations of what constitutes professional competence and competent performance. Even when using the same pieces of information, assessors furthermore differed with respect to inferences about the student as a person as well as a (future) professional. Our findings support the notion that assessors’ reasoning in judgment and decision-making varies and is guided by their mental models of performance assessment, potentially impacting feedback and the credibility of decisions. Our findings also lend further credence to the assertion that portfolios should be judged by multiple assessors who should, moreover, thoroughly substantiate their judgments. Finally, it is suggested that portfolios be designed in such a way that they facilitate the selection of and navigation through the portfolio evidence.

Some assembly required: tracing the interpretative work of Clinical Competency Committees

“…The results of our study support the findings of other studies that suggest that the socially situated professional judgements of CCC members are vital to both the integrity of the evidentiary review process and the determination of decisions on a trainee's progression . In light of this, the effortless data interpretation processes we have described may not be as uncomplicated as they appear to be on the surface, and may operate to obscure limitations of the assessment data on which the decision is based.…”

“…This type of small group decision making is no small task. Committees must sift through large amounts of quantitative and qualitative performance data, interpret their meanings, and create an intelligible composite that reliably communicates the trainee's competencies . This process relies on the expertise and judgement of committee members and their ability to interpret performance evidence and transform it into a decision about trainee competence.…”

“…Concerns around the fairness and transparency of progression decisions have been at the heart of this conversation . The current lack of systematic procedures for evidentiary review, weighting and interpretation and the heterogeneity of practices within committees have been problematised . In response, a variety of strategies have been proposed to standardise CCCs, including best operating practices, faculty training, data visualisation software and data analysis programs …”

Beyond summative decision making: Illuminating the broader roles of competence committees

“…To date, this body of literature has been organised around two themes: (a) developing best practices and forwarding practical advice for the assemblage and operation of CCs, 5‐9 and (b) understanding how the decision‐making work of the CC unfolds 1,10‐14 . The latter literature has a strong emphasis on summative decision making, explicating how these small groups collate and interpret aggregate assessment data to inform decisions about progression and entrustment 1,13,15,16 . Given the high‐stakes nature of these summative decisions, it is unsurprising that they have been subject to significant scholarly interest.…”

Features of Academic Information Perception by Contemporary Higher School Students

“…Анализ специализированной литературы помог выделить ряд подходов и принципов для изучения образовательных технологий и возможности их адаптации в образовательном процессе в условиях распространения клипового мышления [19]. Анализ литературы по методологии обучения в высшей школе на основе компетентностного и практико-ориентированных подходов с позиции коммуникации, трансляции знаний, технических навыков [15], мышления, эмоций, ценностей и отражение их в повседневной практике в целях обеспечения интересов индивида и общества [17], рефлексии и оценки собственных знаний, умений на основе принципа единства профессиональных навыков и личностных характеристик [21], оценки качества образования [14], уровня сформированности компетенций [16], формирования портфолио будущего специалиста [23], а также с позиции современных подходов и форм обучения в высшем образовании [18].…”