Quality Control in the Development and Use of Performance Assessments

Dunbar, Stephen B.; Koretz, Daniel; Hoover, H. D.

doi:10.1207/s15324818ame0404_3

Cited by 215 publications

(127 citation statements)

References 6 publications

Supporting

Mentioning

114

Contrasting

Unclassified

Order By: Relevance

“…Traditionally, reliability is defined in terms of the consistency of measurement over repeated occasions given fixed raters (Dunbar, Koretz, & Hoover, 1991) or the relationship of a single test item to the test as a whole. The first idea might be useful for psychological tests measuring unchangeable traits, but in education, changes in time are expected and even part of educational goals.…”

Section: Wheel Competency Assessmentmentioning

confidence: 99%

The wheel of competency assessment: Presenting quality criteria for competency assessment programs

Baartman

Bastiaens

Kirschner

et al. 2006

Studies in Educational Evaluation

145

103

View full text Add to dashboard Cite

Section: Wheel Competency Assessmentmentioning

confidence: 99%

The wheel of competency assessment: Presenting quality criteria for competency assessment programs

Baartman

Bastiaens

Kirschner

et al. 2006

Studies in Educational Evaluation

145

103

View full text Add to dashboard Cite

“…In generalizability studies, analysis of variance estimates the contribution of multiple sources of error to the score variance. Although some performance assessment studies report that the variance related to the judges makes a negligible contribution to the overall variance (e.g., Shavelson, Baxter, & Gao, 1993), this result does not occur when the assessment task is more complicated (Dunbar, Koretz, & Hoover, 1991).…”

Section: Psychometric Quality Criteriamentioning

confidence: 95%

Cognitive representations in raters' assessment of teacher portfolios

Schaaf

Stokking

Verloop

2005

Studies in Educational Evaluation

View full text Add to dashboard Cite

Portfolios are frequently used to assess teachers' competences. In portfolio assessment, the issue of rater reliability is a notorious problem. To improve the quality of assessments insight into raters' judgment processes is crucial. Using a mixed quantitative and qualitative approach we studied cognitive processes underlying raters' judgments and the reliability of these judgments. Six raters systematically assessed 18 portfolios. The interrater reliability of 12 portfolios was satisfactory. Variance analysis showed slight rater effects. We used the Correspondent Inference Theory (Jones & Davis, 1965) and the Associative Systems Theory (Carlston, 1992; to analyse judgment forms and retrospective verbal protocols. Raters' cognitive representations on the dimensions abstract-concrete and positive-negative were significantly related to the judgments given and to the reliability of these judgments.

show abstract

“…The question of how to value different contributions to the assessment is sometimes described as a trade-off (Dunbar et al, 1991), where either the validity or the reliability is prioritized. On the one hand, reliability is a prerequisite for validity; otherwise, there would be only static, but no signal.…”

Section: Validitymentioning

confidence: 99%

Complement or Contamination: A Study of the Validity of Multiple-Choice Items when Assessing Reasoning Skills in Physics

2017

View full text Add to dashboard Cite

The purpose of this study is to investigate the validity of using multiple-choice (MC) items as a complement to constructed-response (CR) items when making decisions about student performance on reasoning tasks. CR items from a national test in physics have been reformulated into MC items and students' reasoning skills have been analyzed in two substudies. In the first study, 12 students answered the MC items and were asked to explain their answers orally. In the second study, 102 students from five randomly chosen schools answered the same items. Their answers were scored, and the frequency of correct answers was calculated for each of the items. The scores were then compared to a sample of student performance on the original CR items from the national test. Findings suggest that results from MC items might be misleading when making decisions about student performance on reasoning tasks, since students use other skills when answering the items than is intended. Results from MC items may also contribute to an overestimation of students' knowledge in science.

show abstract

Quality Control in the Development and Use of Performance Assessments

Cited by 215 publications

References 6 publications

The wheel of competency assessment: Presenting quality criteria for competency assessment programs

The wheel of competency assessment: Presenting quality criteria for competency assessment programs

Cognitive representations in raters' assessment of teacher portfolios

Complement or Contamination: A Study of the Validity of Multiple-Choice Items when Assessing Reasoning Skills in Physics

Contact Info

Product

Resources

About