Cognitive representations in raters' assessment of teacher portfolios

Abstract

Portfolios are frequently used to assess teachers' competences. In portfolio assessment, the issue of rater reliability is a notorious problem. To improve the quality of assessments insight into raters' judgment processes is crucial. Using a mixed quantitative and qualitative approach we studied cognitive processes underlying raters' judgments and the reliability of these judgments. Six raters systematically assessed 18 portfolios. The interrater reliability of 12 portfolios was satisfactory. Variance analysis showed slight rater effects. We used the Correspondent Inference Theory (Jones & Davis, 1965) and the Associative Systems Theory (Carlston, 1992; to analyse judgment forms and retrospective verbal protocols. Raters' cognitive representations on the dimensions abstract-concrete and positive-negative were significantly related to the judgments given and to the reliability of these judgments.

Is the edTPA the Right Choice for Evaluating Teacher Readiness?

“…Again, it is concerning that only 10 percent of all edTPA submissions are randomly selected to be scored by a second independent scorer (SCALE 2013, 23). While there have been issues reported with interrater reliability in scoring regular holistic TPP portfolios (e.g., Carpenter, Marissa, and Bloom 1995;Derham and Diperna 2007;Koretz 1998;Oskay, Schallies, and Morgil 2008;Yao et al 2008), there is a strong body of research illustrating how these issues can be rectified (e.g., Baume and York 2002;Draves 2009;Johnson, McDaniel, and Willeke 2000;Meeus, Van Petegem, and Engels 2009;van der Schaaf, Stokking, and Verloop 2005). Recent research on music education TPPs illustrates that the content-specific pedagogical knowledge of the raters is critical when evaluating electronic portfolios for the connections between the artifacts within portfolio (Parkes 2013).…”

Adaptación y Pilotaje de un Portafolio Para Evaluar Prácticas de Evaluación de Aprendizajes en el Aula en Profesoresde Matemática de Segundo Ciclo Básico

“…El portafolio como herramienta de evaluación ofrece una mirada más completa y global del desempeño docente e incluye una mayor cantidad de fuentes de información, lo que hace que la labor interpretativa de los correctores sea una parte fundamental (van der Schaaf et al, 2005). Los correctores, antes de evaluar, se hacen una imagen mental de la evidencia, para luego -influidos por su comprensión de los criterios de evaluación, las experiencias previas, el contexto, las condiciones del proceso de corrección y la importancia que le asignan a diferentes evidencias del portafolio-asignan un puntaje para cada dimensión (Heller, Sheingold & Myford, 1998).…”

Assessors’ approaches to portfolio assessment in Assessment of Prior Learning procedures

“…Research in the context of teacher education has shown that objective and unambiguous portfolio assessment is difficult (Tigelaar et al 2005;Van der Schaaf, Stokking, and Verloop 2005), and how assessors can gauge the level of candidates' prior learning is unclear (Trowler 1996). Although portfolio assessment should meet quality requirements such as reliability and validity, in practice it is often difficult to sufficiently address these criteria.…”

“…According to Heller, Sheingold, and Myford (1998), the process of portfolio assessment consists of three essential components: evaluating individual texts in the portfolio one at a time, evaluating across texts in the portfolio and articulating a rating, weighing relative amounts or importance of evidence as needed. Van der Schaaf, Stokking, and Verloop (2005) showed, in the context of teacher education, that judgements were influenced by previous ratings and experiences, and that despite agreeing on ratings there remains a difference in portfolio interpretation. In addition, intuition, as a domain-specific competence to reach an appropriate decision, is mentioned as a crucial component of a decision-making process by professionals (Harteis and Gruber 2008).…”