Developing Performance Standards for Teacher Assessment by Policy Capturing

Cognitive representations in raters' assessment of teacher portfolios

“…This tendency could be considered as undesirable, however, since it is possible that one criterion is more important than another (and thus should get more weight). In one of our two previous studies (Van der Schaaf et al, 2003) we let the raters formulate a panel policy in which they decided that some criteria should be weighted more heavily than others. The tendency to use unweighted means is strong, however, as the holistic scores and the unweighted mean analytic scores corresponded highly although the raters were instructed and trained in weighting the scores according to the policy.…”

“…This can only be realized when raters interpret the portfolio artefacts and the standards in completely the same way. As people always may differ in their interpretations (Huot, 1993;Kane, 1992;Van der Schaaf et al, 2003), these conditions cannot be guaranteed. Therefore, interrater reliability in terms of percentage agreement can only be used as an indicator of reliability.…”

“…In two previous studies we developed both content standards (criteria) and performance standards (cut-off scores) for teaching research skills in uppersecondary (ages 16--18) social science education (Van der Schaaf, Stokking, & Verloop, 2003. Teaching research skills is a recent Dutch curricular innovation which fits well with the change in many countries towards more constrnctivist views on learning and developing students' skills in an active, self-directed and collaborative way,…”

“…We chose to use external raters, that is raters not working in the same schools as the teachers to be assessed, in order to avoid raters biasing their perceptions or judgments on episodic memories and local orientations. We selected raters who successfully participated in the two previous studies (described above) in which content standards, performance standards, and a portfolio assessment procedure were developed (Van der Schaaf et al, 2003Schaaf et al, , 2004. So, the raters were familiar with the project and they subscribed to the standards and the assessment procedure (prerequisites for any assessment process to be suecesful).…”

“…In a second previous study (Van der Schaaf et al, 2003), using policy capturing we developed the performance standards (the cut-off scores that indicate how good teachers should meet the content standards for a sufficient result). Policy capturing uses multiple regression procedures to develop the cut-off scores and to produce a best fitting equation or policy towards the weighting of the criteria and the specific judgment model.…”